Image processing apparatus, an image processing method, a medium on which an image processing control program is recorded, an image evaluation device, an image evaluation method and a medium on which an image evaluation program is recorded

ABSTRACT

In image processing according to the prior art, the important part of photographic image data (referred to herein as the object) could not be determined and therefore required human participation.  
     A computer  21  which is the core of image processing calculates an edginess which is an image variation from a differential value of data for adjacent picture elements in a step SA 110,  and determines object picture elements by selecting only images with a large variation in steps SA 120,  SA 130.  As optimum parameters for contrast correction and lightness compensation are calculated from image data for object picture elements in steps SA 310 -SA 330,  image processing indicators based on object picture elements are determined, and optimum image processing can be performed automatically. After summing a luminance distribution for each area of the image, which is a feature amount, while uniformly selecting picture elements in a step SB 110,  a reevaluation is performed by a weighting determined for each area in a step SB 120,  and a luminance distribution strongly influenced by the luminance distribution of the photographed object is thus obtained with uniform sampling. After determining the intensity of this luminance distribution in steps SB 130 -SB 150,  the image data is converted in a step SB 160 , and image processing can therefore be performed with optimum intensity while reducing the processing amount.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] This invention relates to an image processing method wherein anoptimum image processing is performed automatically on photograph imagedata such as digital photograph image, and the image is evaluated, to animage processing apparatus, a medium on which an image processingcontrol program is recorded, an image evaluation device, an imageevaluation method, and a medium on which an image evaluation program isrecorded. 2. Description of the Prior Art

[0003] Various kinds of image processing may be performed on digitalimage data, i.e., in which processing: contrast may be increased; colormay be corrected; or lightness may be corrected. This image processingcan usually be performed with a microcomputer. An operator confirms theimage on a monitor, the necessary image processing is selected, andimage processing parameters are determined.

[0004] In recent years various types of image processing techniques havebeen proposed, and are now having considerable impact. However, a humanoperator is still required when it is a question of which technique toapply, and to what extent it should be used. This is because it wasotherwise impossible to determine which digital image data had to besubjected to image processing. For example, in the case of imageprocessing to correct the lightness of an image, the screen is madelighter if it is dark on the whole, and is made darker if it is toolight.

[0005] Now, consider the case of a photographic image of a person filmedat night, where the background is near to pitch-darkness but the personin the picture has been well photographed. If this photograph isautomatically corrected, it is attempted to make the image brighter dueto the fact that the background is pitch black, so the final imageappears as if the photo was taken in the daytime.

[0006] In this case, if a human operator is involved, he pays attentiononly to the person in the picture. If the image of the person is dark,it would be made a little brighter, conversely darkening would beselected if the effect of flash, etc., was too bright.

[0007] Hence, there was a problem in the prior art in that a humanoperator had to participate to determine the important part (referred tohereafter as the “object”) of a photographic image.

[0008] However, even when the importance of the image is evaluated bysome technique, the determination process is performed in pictureelement units, and varying the importance in re,al time causes anincrease in computation.

SUMMARY OF THE INVENTION

[0009] It is therefore an object of this invention, which was conceivedin view of the aforesaid problems, to provide an image processing methodwhich permits an important part of a photographic image such as adigital photograph image to be detected, and an optimum image processingto be automatically selected, to provide an image processing apparatus,and to provide a medium on which an image processing control program isrecorded.

[0010] In order to achieve the aforesaid object, this invention is animage processing apparatus into which photographic image data comprisingdot matrix picture elements is input, and which performs predeterminedimage processing on the input data. The apparatus comprises an imageprocessing acquiring unit which acquires the aforesaid photographicimage data, an image processing indicator specifying unit which performspredetermined summation processing on picture elements and specifies animage processing indicator based on the acquired image data, and aprocessing unit which determines image processing contents based on thespecified image processing indicator, wherein the aforesaid imageprocessing indicator specifying unit comprises an object determiningunit which determines picture elements having a large image variationamount to be those of the object, and the aforesaid processing unitdetermines image processing contents based on image data for pictureelements determined to be those of the object and performs imageprocessing on the determined contents.

[0011] Herein, it is assumed that in the case of an image of aphotographic image of a person, the person is usually photographed inthe center of the field. Therefore, the person is brought into focus togive a sharp image. When the image is sharp, the outline part becomesclear, and the amount of image variation becomes large. As a result,there is an extremely high possibility that there will be no error if itis assumed that picture elements with a large image variation amount arethose of the original object which has been brought into focus.

[0012] In the invention thus comprised, in the image processingindicator specifying unit based on photographic image data comprisingdot matrix picture elements acquired in the image data acquiring unit,predetermined summation processing is performed on picture elements.This summation processing may take various forms, and the imageprocessing indicator is basically specified based on the summationresult. In the processing unit, the image processing contents aredetermined based on the specified indicator, and the determined imageprocessing is performed.

[0013] This means that useful information about the corresponding imagecan be obtained by performing summation processing on dot matrixphotographic image data, and image processing is performed on the data.In this way, the image processing indicator is specified by actualphotographic image data, so optimum image processing can be performedeven without human intervention.

[0014] The photographic image data comprises dot matrix picture elementsand image processing is performed in picture element units, but first,in the object determining unit, picture elements having a large imagevariation are determined to be those of the object. Image processingcontents are then determined in the processing unit based on image datafor picture elements determined to be those of the object, and imageprocessing is performed based on the determined contents.

[0015] Therefore, according to this invention, the determination of theobject, which in the past required human participation, can be automatedby determining the object as picture elements with a large imagevariation. The invention therefore provides an image processing methodwhereby optimum image processing can be performed by suitably modifyingthe image processing contents according to the object.

[0016] This method for specifying an image processing indicator fromactual photographic data may of course be applied not only to a realdevice but also to a system on the method. In such a sense, thisinvention is also an image processing method wherein photographic imagedata comprising dot matrix picture elements is input, and predeterminedimage processing is performed, this image processing method comprisingan image data acquiring step for acquiring the aforesaid photographicimage data, an image processing indicator specifying step for performinga predetermined summation processing on picture elements based on thisacquired image data, and specifying an image processing indicator, and aprocessing step for determining image processing contents based on thespecified indicator, and performing image processing, wherein theaforesaid image processing indicator specifying step comprises an objectdetermining step wherein picture elements having a large image variationamount are determined to be those of the object, and wherein in theaforesaid processing step, image processing contents are determinedbased on image data for picture elements determined to be those of theobject, and image processing is performed based on the determined imageprocessing contents. In this case, the apparatus offers all theattendant benefits and advantages of the invention.

[0017] This apparatus for determining an object and performing imageprocessing method may be implemented by a stand-alone apparatus asmentioned above, or may be incorporated in another instrument whichcomprises such an apparatus. In other words, the scope of this inventioncovers various forms of implementation. It may also be implemented byhardware or software, and can be modified as necessary.

[0018] When the apparatus for implementing the concept of this inventionis implemented by software, the invention applies equally to media onwhich this software is recorded and which can be used in exactly thesame way. In this sense, this invention is also a recording mediumwhereon an image processing control program is recorded for inputtingphotographic image data comprising dot matrix picture elements by acomputer, and which performs predetermined image processing on the inputdata. The control program comprises an image processing indicatorspecifying step for acquiring the aforesaid photographic image data, animage processing indicator specifying step for performing predeterminedsummation processing on picture elements and specifying an imageprocessing indicator, and a processing step for determining imageprocessing contents based on the specified image processing indicator,wherein the aforesaid image processing indicator specifying stepcomprises an object determining step which determines picture elementshaving a large image variation amount to be those of the object, and inthe aforesaid processing step, image processing contents are determinedbased on image data for picture elements determined to be those of theobject, and image processing is performed on the determined contents. Inthis case, the recording medium offers all the attendant benefits andadvantages of the invention.

[0019] The recording medium may of course be a magnetic recordingmedium, an optomagnetic recording medium, or any kind of recordingmedium which may be developed in the future. It will of course beunderstood that the medium may be a first copy or second copy, and thata telecommunication line may also be used to supply the program. In thiscase, there is no difference regarding the application of the invention.There is also no difference if the program is written on a semiconductorchip.

[0020] There is no difference as regards the concept of the inventioneven if one part is software, and one part is implemented with hardware,or when it is in such a form that one part is stored on a recordingmedium which can be read when necessary.

[0021] Photographic image data means image data obtained when it isattempted to take a photograph of a real object. Image processing triesto correct images by comparing the images with the real objects fromwhich they were acquired. The invention therefore applies not only tonatural objects but also to manmade ones. More specifically, thisincludes image data read by a scanner, or image data captured by adigital camera.

[0022] Various techniques may be employed to determine the variation ofpicture elements in the object determining step. A further object ofthis invention is to give a specific example of this.

[0023] In the image processing method provided by this invention, in theaforesaid object determining step, the amount of variation of pictureelements is determined based on a difference between adjacent pictureelements.

[0024] Hence according to this invention, in the object determining unitwhere an image variation amount is determined, the determination isperformed based on a difference of image data between adjacent pictureelements. When there is a fixed interval between picture elements as inthe case of a dot matrix, the difference of data between adjacentpicture elements is directly proportional to a first order differential.This difference can be taken as the variation amount of the image. Inthis case the difference is regarded as the magnitude of a vector, andthe vector may also be constructed by considering adjacent directions.

[0025] According to this invention, only the difference of image databetween adjacent picture elements is found. Computation is thereforeeasy, and the processing amount for object determination can be reduced.

[0026] The determination of an object is of course not limited to thistechnique, and it is a further object of this invention to provide otherexamples.

[0027] According to the image processing apparatus provided by thisinvention, in the aforesaid object determining unit, the criterion fordetermining whether or not there is a large image variation amountchanges according to the position of the image.

[0028] In the case of a photograph for example, a person is oftenphotographed in the center. In this case it may be said that in order todetermine image processing content, the picture elements to bedetermined as the object should be selected from the central part of thefield. However, it may be said that whether or not there is a largevariation in the image depends on a difference from a comparison value,and there is no reason why such a value always has to be constant.

[0029] Therefore, to determine whether or not there is a large imageamount according to this invention, in the object determining unit, thiscriterion is altered depending on the position of the image, thecriterion for each position being compared with the image variationamount of each picture element.

[0030] Hence, according to this invention, the assessment of imagevariation changes depending on the position of the image, and a highlyflexible determination which considers image composition is thuspossible.

[0031] The criterion can be altered in various ways. As one example, acertain trend may be ascertained, or alternatively, a trend which causesa change may be read from the image.

[0032] A further object of this invention is to provide an example ofthe former.

[0033] In the image processing apparatus provided by this invention, inthe object determining unit, the aforesaid criterion is set lower forthe central part than for the edges of the image.

[0034] By setting the criterion lower for the center than for the edges,it is easier to determine the center part of the image as the objecteven if the variation amount at the center and at the edges isapproximately the same. Therefore if there is an image of a person inthe central part, the picture elements of this person will be determinedas the object more frequently.

[0035] According to this invention, a determination can be made whichgives more weight to the center area of a photograph, and a large amountof image data can be effectively processed.

[0036] A further object of this invention is to provide an example ofthe latter when the criterion is varied.

[0037] According to the image processing apparatus provided by thisinvention, in the aforesaid object determining unit, the above criterionis based on the distribution of the aforesaid image variation amount atdifferent points on the image.

[0038] Hence according to this invention, in the object determiningunit, the distribution of image variation is found in each part of theimage, and the aforesaid criterion is determined after finding thisdistribution. Subsequently, a comparison is made with this criterion todetermine whether or not the picture elements are those of the object.

[0039] According to this invention, as the object is determined takingaccount of the distribution of image variation for picture elements, theimage data can be treated flexibly.

[0040] When the criterion is determined based on distribution, it may beconsidered that there is a high possibility of finding the object in apart where there are many picture elements with a large variationamount, and the criterion maybe set low.

[0041] Alternatively, basic setting patterns may first be preparedaccording to a variation distribution pattern, and a basic settingpattern may then be chosen based on the detected distribution pattern.

[0042] At the same time, assuming that the image processing indicatorspecifying unit comprises such an object determining unit, the imageprocessing contents may be determined based on image data which isdetermined to be that of the object, and image processing may then beperformed on the determined contents, there being no limitation on thespecific processing method employed. For example, a luminancedistribution of picture elements determined to be those of the object isfound, and if the luminance distribution range is enlarged in apredetermined proportion when the luminance distribution is narrow,image processing to increase contrast is performed. If the luminancedistribution of the object seems dark on the whole, a correction may bemade to make it lighter. The color distribution of picture elementsdetermined to be those of the object is found, and it is determinedwhether or not the grey balance is off. If it seems to be off, tonecurves are used to modify the grey balance.

[0043] Hence, the importance of the image has an effect even if theimage data is summed in order to specify the image processing indicator.However, even if the importance of the image is determined by sometechnique, the work is carried out in picture element units, so varyingthe importance of an image in real time implies an increase ofcomputational amount.

[0044] A further object of this invention is to consider the importanceof photographic image data such as digital photograph images inrelatively simple terms, and perform optimum image processingautomatically.

[0045] The image processing apparatus provided by this invention is anapparatus for inputting photographic image data comprising dot matrixpicture elements, and performing predetermined image processing. Thisimage processing apparatus comprises an image data acquiring unit foracquiring the aforesaid photographic image data, an image processingindicator specifying unit which performs a predetermined summationprocessing on picture elements based on this acquired image data andspecifies an image processing indicator, and a processing unit whichdetermines image processing contents based on the specified indicatorand performs image processing. The aforesaid image processing indicatorspecifying unit comprises a feature amount uniform sampling unit whichdetermines an image processing intensity by uniformly sampling a featureamount over a whole screen, and a feature amount weighting reevaluationunit which reevaluates the feature amount sampled in the feature amountsampling unit with a predetermined weighting. In the aforesaid imageprocessing unit, the image processing intensity is determined based onthe reevaluated feature amount, and image processing is performed withthe determined intensity.

[0046] According to the invention having the above construction,photographic image data comprises dot matrix picture elements, and inthe feature amount uniform sampling unit, the feature amounts of pictureelements are uniformly sampled over the whole screen. In the featureamount weighting reevaluation unit, the feature amounts that are sampledin this feature amount uniform sampling unit are reevaluated with apredetermined weighting. Then, in the processing unit, the imageprocessing intensity is determined based on the feature amounts thathave been reevaluated in this way, and image processing is performed.

[0047] In other words, as the sampling is uniform over the whole screenand a predetermined weighting is applied after sampling, the featureamounts obtained as a result are different from what is obtained byuniform sampling over the whole screen.

[0048] According to this invention the sampling in the sampling stage isuniform over the whole screen, so the computational amount is not toohigh. At the same time, by applying a predetermined weighting aftersampling, irrelevant evaluation is not made as it would be if thepicture elements were merely sampled uniformly without weighting. Theinvention therefore provides an image processing apparatus in whichoptimum image processing can be performed automatically.

[0049] It will be understood that the technique of performing a uniformsampling in the sampling stage and applying a predetermined weightingthereafter, may be applied not only to a real device but also to asystem in both of which cases it has all the attendant benefits andadvantages of this invention. As a specific example of the concept ofthis invention, when the image processing apparatus is implemented interms of software, there naturally exist recording media on which thesoftware is recorded which can be used to perform the function of theinvention.

[0050] The feature amount uniform sampling unit uniformly samplesfeature amounts over the whole screen, for determining the imageprocessing intensity. For this purpose, all picture elements over thewhole screen can be sampled, but it is not necessary to sample all ofthe picture elements if the sampling is uniform.

[0051] A further object of this invention is to provide an example ofthe latter case.

[0052] According to the image processing apparatus of this invention, inthe aforesaid feature amount uniform sampling unit, the aforesaidfeature amounts are sampled for selected picture elements afteruniformly thinning out the picture elements according to predeterminedcriteria.

[0053] According to this invention, by thinning out the picture elementsaccording to predetermined criteria, the number of picture elements tobe processed is reduced, and the aforesaid feature amounts are sampledfrom the remaining elements.

[0054] Herein, the term “uniform thinning” comprises the case wherepicture elements are selected at a fixed interval, and the case wherethey are selected at random.

[0055] According to this invention, as the picture elements are thinnedout when the feature amounts are uniformly sampled, the processingamount is reduced.

[0056] The sampled feature amounts are reevaluated by a predeterminedweighting in the feature amount weighting reevaluation unit. The sampledfeature amounts are in picture element units, but the weighting can beapplied either to picture element units or to suitable aggregates ofpicture elements.

[0057] A further object of this invention is to provide an example ofthe latter case.

[0058] According to the image processing apparatus of this invention, inthe aforesaid feature amount uniform sampling unit, feature amounts aresampled in area units that are divided according to predeterminedcriteria, and in the aforesaid feature amount weighting reevaluationunit, a weighting is set for each area and the feature amounts are thenreevaluated.

[0059] The invention as formulated hereabove assumes weightings in areaunits of the image that are divided according to predetermined criteria.In the feature amount uniform sampling unit, feature amounts are sampledin these area units, while in the aforesaid feature amount weightingreevaluation unit, the feature amounts are reevaluated with weightingsset for each area.

[0060] The division of these areas may always be constant, or it may bemade to vary for each image. In this latter case the division method maybe changed according to the contents of the image.

[0061] According to this invention, as the weighting is made to vary foreach area, the computation is relatively simple.

[0062] Any type of weighting technique can be employed provided thatreevaluation is performed without merely performing uniform sampling.

[0063] A further object of this invention is to provide an example ofthis.

[0064] According to the image processing apparatus of this invention, inthe aforesaid feature amount weighting reevaluation unit, the aforesaidweighting is made to vary by a correspondence relation determined by theposition of picture elements in the image.

[0065] In the case of a photograph, the person is usually in the center.Therefore, by weighting the central part of the image more heavily afteruniformly sampling feature amounts from the whole image, the featureamounts sampled from picture elements relating to the person areevaluated to be larger.

[0066] When for example according to the invention thus comprised, theweighting of the central part of the image is heavier and the weightingof the surroundings is lighter, in the feature amount weightingreevaluation unit, the position of picture elements in the image isdetermined, and a reevaluation is made using a weighting which variesaccording to this position.

[0067] Hence according to this invention, as the weighting is determinedaccording to the position of picture elements, the computation isrelatively simple.

[0068] The weighting technique is of course not limited to this method,and a further object of this invention is to provide other examples.

[0069] According to the image processing apparatus provided by thisinvention, in the aforesaid feature weighting reevaluation unit, theimage variation amount is found, and a heavier weighting is given toparts where the image variation amount is larger.

[0070] In the invention thus comprised, the image variation amount isfound before performing reevaluation in the feature amount weightingreevaluation unit. The image variation amount is also known as imagesharpness, and as the outline is sharper the better the focus, thevariation is large where the image is in focus. On a photograph, thepart which is in focus is the subject, and the part which is not infocus is considered to be the background. Therefore, places where thereis a large image variation are considered to correspond to the subject.In the feature amount weighting reevaluation unit, the same result assampling a large number of feature amounts is obtained by applying heavyweighting to parts where there is a large image variation.

[0071] According to this invention, as the weighting is varied dependingon image sharpness, different targets can be precisely identified andfeature amounts can be sampled for different images.

[0072] As another example of a weighting technique, in the weightingreevaluation unit of the image processing apparatus of the invention,the chromaticity of picture elements is found, a number of pictureelements is found for which the chromaticity lies within thechromaticity range of the target for which it is desired to sample afeature amount, and heavier weighting is applied to parts where thereare many of these picture elements.

[0073] Hence according to this invention, in the feature amountweighting reevaluation unit, the chromaticity of picture elements isfound. In image processing, an object can sometimes be specified by aspecific chromaticity. For example, there is no reason why a personcould not be identified by looking for skin color, but it is difficultto specify skin color as color data also contain luminance elements.Chromaticity on the other hand represents an absolute proportion of acolor stimulation value, and it is not controlled by luminance.Therefore an image of a person could be determined if the chromaticitywas within a specified range that can be taken as indicative of skincolor. This reasoning may of course also be applied to the green of thetrees or the blue of the sky.

[0074] As specified objects can be sorted by chromaticity according tothis invention, different targets may be precisely sampled depending ontheir images and feature amounts sampled.

[0075] Therefore in a feature amount weighting reevaluation unit, whenthe chromaticity found for picture elements is within a chromaticityrange for a target from which it is intended to sample feature amounts,plural picture elements are counted. When the number of picture elementsis large, this part of the image is determined to be the target, heavyweighting is applied, and a large feature amount is sampled from thetarget.

[0076] This weighting technique is not necessarily the only alternative,and it is a further object of this invention to provide a suitableexample of overlapping methods.

[0077] According to the image processing apparatus of this invention, inthe aforesaid feature amount weighting reevaluation unit, temporaryweightings are applied based on a plurality of factors, and thesefactors are then added according to their degree of importance to givefinal weighting coefficients.

[0078] According to the invention as thus comprised, in the featureamount weighting reevaluation unit, temporary weighting coefficients arefound separately based on a plurality of factors, and the weightings areadded according to their degree of importance so as to reevaluate thesampled feature amounts as final weighting coefficients. Therefore, itmay occur that even when a large weighting is assigned by one weightingmethod in the evaluation stage, if the method does not have a largeimportance, the final weighting which is assigned is not large.Moreover, it may occur that even if there is a large difference betweenweighting methods, image parts which are evaluated to have an average orhigher weighting also have a large final weighting.

[0079] According to this invention, plural weighting techniques aresuitably combined so that a suitable feature amount evaluation can beperformed.

[0080] Therefore if the image processing indicator specifying unititself comprises plural forms, it is not necessary to perform imageprocessing with only one of these forms.

[0081] However even if there are some cases where it is desirable toperform image processing using the feature amount of the object, thereare other cases where it is desirable to perform image processing usingan average feature amount for the whole photographic image. For example,when a photograph is taken of a person, the person may not always be thelightest (highlighted) part of the picture. Therefore if attention ispaid only to the person in the picture and contrast is increased, thehighlighted part of the background will be too white. In this case, abetter result would be obtained by paying attention to the wholephotographic image.

[0082] Hence when image processing is performed, it is still necessaryto select an optimum feature amount.

[0083] It is a further object of this invention to automatically selectan optimum feature amount according to image processing technique.

[0084] In the image processing apparatus according to this invention,photographic image data comprising dot matrix picture elements is input,and predetermined image processing is performed. This image processingapparatus comprises an image data acquiring unit for acquiring theaforesaid photographic image data, an image processing indicatorspecifying unit for performing a predetermined summation processing onpicture elements based on this acquired image data, and specifying animage processing indicator, and a processing unit for determining imageprocessing contents based on the specified indicator, and performingimage processing. The aforesaid image processing indicator specifyingunit comprises an evaluation unit for obtaining a feature amount byinputting the aforesaid photographic data, summing the image data forall picture elements, and obtaining a feature amount according to pluralpredetermined evaluation criteria. In the aforesaid processing unit, theimage data can be converted by plural techniques, and the featureamounts obtained in the aforesaid evaluation unit used according to theparticular technique.

[0085] According to this invention, image data from a photographic imagecomprising dot matrix picture elements is input in this manner, and afeature amount is obtained according to plural evaluation criteria bysumming image data for picture elements in the evaluation unit. In theprocessing unit, in converting the image data by plural techniques, thefeature amounts obtained in the evaluation unit according to eachtechnique are then used to convert the data depending on the technique.

[0086] Specifically, although there are cases where the image data isbest converted using a feature amount centered on the object such as inlight/dark correction, there are other cases where image data is betterconverted using a feature amount centered on the whole image such aswhen contrast is increased, and the image data conversion may beperformed by suitably selecting these plural feature amounts.

[0087] According to this invention, when feature amounts are obtainedaccording to plural evaluation criteria and image processing isperformed by plural methods, the feature amounts used depend on themethod, so image processing may be performed based on an optimumevaluation criterion.

[0088] When image data is converted, the feature amounts should be suchthat they can be used to identify the features of the image, and thereis no need to specify the type of image. For example this also includesindicators such as luminance histograms which identify whether the imageis to be considered as light or dark, there being no need to obtain theidentification result that the image is light or dark. Apart fromlightness, the indicator may of course specify whether or not the imageis sharp, or it may be an indicator to identify vividness.

[0089] There is also no particular limitation on the way in which theevaluation unit and processing unit are applied. For example, assumingthat plural image processing methods are used, plural feature amountsmay be obtained and stored according to plural evaluation criteria, andthe image data converted by suitably selecting the feature amount in theprocessing unit as necessary. As another example, image data for pictureelements may be summed by a predetermined criterion in the evaluationunit so as to obtain a suitable feature amount on each occasion thatimage processing is performed in the aforesaid processing unit.

[0090] It will of course be understood that these plural imageprocessing methods based on feature amounts obtained by plural differentevaluation criteria, may also be applied not only to an actual devicebut also to a system both of which are then a valid form of theinvention. When the image processing methods are implemented by softwareas specific examples of the concept of the invention, there naturallyexist media on which the software is recorded which then offer all theattendant advantages thereof.

[0091] The evaluation criterion used to obtain feature amounts in theaforesaid evaluation unit will depend on the image processing that is tobe performed, and while there are some cases where it is desirable toconcentrate on the object for image processing, there are some caseswhere it is not as described above.

[0092] It is a further object of this invention to provide an example ofthe former case.

[0093] In the image processing apparatus according to this invention,the aforesaid evaluation unit comprises an evaluation unit wherein anobject in a photographic image is sampled, and image data for pictureelements of this object is summed to obtain a feature amount, and in theaforesaid processing unit, in one processing method, the feature amountobtained from object picture elements is used when the feature amountfor the central part of the image data is used.

[0094] According to the invention thus comprised, when image data isconverted based on the feature amount of the central part of the imagedata in the aforesaid processing unit, the object in the photographicimage is sampled in the aforesaid evaluation unit, and the featureamount is obtained by summing image data for object picture elementsaccording to predetermined criteria.

[0095] Herein, the central part of the image data has the followingmeaning. For example, when it is determined whether a given photographis light or dark, it is easily appreciated that the determination canconveniently be based on the intermediate density of the image. Thisintermediate density may also be referred to as a median in a luminancedistribution, i.e. the center of the luminance distribution, and in thissense it is referred to as the central part of the image data. Then, ifthere is an object in the photographic image, it may be said that thereis a definite necessity to perform light/dark correction in line withthe lightness of this object.

[0096] This invention is suitable for the case when image processing isperformed based on the feature amount of the central part of the imagedata.

[0097] Any of the aforesaid techniques may be applied as the basictechnique for sampling the object. As an example, in the aforesaidevaluation unit, picture elements for which there is a large variationof image data between adjacent picture elements are sampled as theobject. When picture elements are aligned at a fixed interval apart asin the case of a dot matrix image, the difference of image data betweenadjacent picture elements is proportional to a first order differential.This difference may be determined as the image variation amount. In thiscase, the difference may be regarded as the magnitude of a vector, andthe vectors constructed taking account of adjacent directions. If thisis done it is sufficient to determine the difference of image data foradjacent picture elements, computing is easy, and the processing fordetermining the object is reduced.

[0098] As another example, in the aforesaid evaluation unit, pictureelements for which the chromaticity is within a predetermined range maybe sampled as the object. In this case, in the aforesaid evaluationunit, the chromaticity of picture elements is found. The chromaticityrepresents an absolute proportion of a color stimulation value, and itis not affected by lightness. Therefore the object in the image can beseparated by possible range of chromaticity. For example, there is thechromaticity range for skin color, or the chromaticity range for thegreen of the trees. As this can be said for chromaticity, in theaforesaid evaluation unit, picture elements for which the chromaticitylies within a predetermined range are sampled as the object. In thisway, an object can be determined by its chromaticity, and the object maybe sampled without depending on the lightness or darkness of the object.

[0099] On the other hand as an example of image processing not concernedonly with the object, the aforesaid evaluation unit comprises anevaluation criterion wherein picture elements of the aforesaid imagedata are uniformly sampled and summed so as to obtain a feature amount,and in the aforesaid processing unit, in one processing method, thefeature amount obtained by the aforesaid uniform sampling is used whenan average feature amount of the photographic image is used. In thiscase, when image data is converted based on the average feature amountin the aforesaid processing unit, the feature amount is obtained byuniformly sampling picture elements of the image data according topredetermined evaluation criteria. Of course, the summation may beperformed on all picture elements of the photographic image, but it maybe said that is no advantage as the processing amount increases. Hence,it is convenient to perform image processing based on the averagefeature amount of the photographic image, e.g. saturation correction.

[0100] As another example of image processing which is not concernedonly with the object, the aforesaid evaluation unit comprises anevaluation criterion wherein picture elements of the aforesaid imagedata are uniformly sampled and summed to obtain a feature amount, and inthe aforesaid processing unit, in one image processing method, thefeature amount obtained by uniform sampling is used when the edges of afeature amount distribution of the photographic image are used. In thiscase, in the aforesaid processing unit, it is assumed that the ends ofthe feature amount distribution obtained in the aforesaid evaluationunit are used. For example, to increase the contrast, image processingis performed to find the luminance distribution, and the edges of thisluminance distribution are widened, but if the luminance distribution ofthe object were used in this case, other highlighted parts appear white.Therefore in this case, in the aforesaid evaluation unit, pictureelements of image data are uniformly sampled according to predeterminedcriteria, and summed to obtain the feature amount. This is suitable forimage processing using the edges of a feature amount distribution in anactual photographic image, e.g. for increasing contrast.

[0101] In the above, a continuous sequence of processes is performedcomprising predetermined analysis of the image and image processing byspecifying image processing indicators, but the analysis result itselfis also useful.

[0102] It is a further object of this invention to provide an imageevaluation device wherein it is easier to use an image evaluation resultwhich is an analysis result done.

[0103] In the image evaluation device offered by this invention,photographic image data comprising dot matrix picture elements is input,the image data for all picture elements is summed according topredetermined criteria, and is the image evaluation device which isbased summation result, and evaluate image, and the image is evaluatedbased on the summation results. There are plural evaluation criteria forthe aforesaid summation results, and the evaluation results are combinedwith a predetermined weighting based on these evaluation criteria.

[0104] According to the invention as thus comprised, the evaluationmethod assumes that photographic image data comprising dot matrixpicture elements is input, the image data is summed for pictureelements, and the image is evaluated based on the summation results.Herein, there are plural evaluation criteria for these summationresults, and the evaluation results are combined with a predeterminedweighting based on the evaluation criteria.

[0105] In other words, although some evaluation criteria are suitablefor evaluating images where a sharp image is the object such as in thecase of portrait, other criteria are suitable for evaluating imageswhere the background is the important object. A general evaluation maybe made by suitably combining plural evaluation criteria in parallel andvarying the weightings.

[0106] As described hereabove, as this invention gives a generalevaluation by varying the weightings of plural evaluation criteria, itprovides an image evaluating device which can be flexibly adapted toimage feature determination.

[0107] Naturally, the concept of this invention for image evaluation bythe above techniques comprises many forms. Specifically, it compriseshardware and software, various modifications being possible as may beconvenient. When the concept of the invention is implemented by imageprocessing software, there naturally exist recording media on which thesoftware is recorded which can be used to perform the function of theinvention. Moreover, these image evaluating techniques may be applied toan image evaluating device and its system running on software media.

[0108] Various techniques may be employed to achieve the same object inapplying plural evaluation criteria to summation results. For example,all picture elements may be summed by weighting with differentevaluation criteria, but it will be appreciated a large amount ofprocessing is involved when the summation is applied to all pictureelements. Hence, as described above, the picture elements are firstsampled based on plural evaluation criteria, summed, and the summationresults combined with a predetermined weighting. In this case, the imagedata are sampled prior to summation, plural criteria are used by varyingthe criteria applied to the sampling, and the weighting of the summationresults is then adjusted before combination. An evaluation can thereforebe made with different weightings on the results based on pluralevaluation criteria. Due to this sampling of image data, pluralevaluation criteria may be employed according to the sampling method.

[0109] As one evaluation criterion, the data may of course be sampleduniformly and summed. In this case, the image data is uniformly thinnedand the whole image is considered, which makes this a suitable criterionfor determining scenic photographs, etc. In this way an optimumcriterion can be used while reducing the processing amount.

[0110] As an example of a criterion which can be applied whether or notsampling is used, is the evaluation of picture elements which have alarge variation relative to adjacent picture elements with a heavierweighting. In this case, the image variation of picture elements fromadjacent elements is detected, and clear image parts with a largevariation are given a heavier weighting in the summation. If this isdone, as image parts with a large variation are often parts of thephotograph which are clearly in focus, an image evaluation which weightsimportant parts of the image can be performed.

[0111] This criterion places more emphasis on the sharp parts of theimage, and it is therefore naturally suitable for the determination ofhuman images. Herein, image parts with a large variation may beevaluated either by introducing weighting as the picture elements aresummed, or by summing only picture elements with a large variation. Theweighting used in the evaluation is not necessarily fixed, but may alsobe allowed to vary according to the criterion. In this case, by varyingthe weighting for different evaluation criteria, an overall evaluationresult for the image can be deduced. Moreover various approaches arepossible, e.g. weightings may be varied individually so as to generateplural combinations which are then selected. In this way, by modifyingthe weighting for plural criteria, a more flexible evaluation can bemade.

[0112] Instead of an operator varying the weighting, this can be donebased on the image data itself. As an example of this, the weighting ofthe evaluation results may be varied based on the criteria. In thiscase, results are obtained according to various criteria, and theweighting is modified in view of the suitability of the criteria in thelight of the results. As the results are used to vary the weighting, thework involved in the evaluation is less.

[0113] Various techniques may also be used to modify the weighting ofthe criteria using the results. For example, if it is determined whetheror not the image data for picture elements should be sampled accordingto one criterion, the number of these picture elements may be taken as acriterion and the weighting increased when the number of pictureelements is large.

BRIEF DESCRIPTION OF THE DRAWINGS

[0114]FIG. 1 is a block diagram of an image processing system in whichan image processing apparatus according to one embodiment of thisinvention is applied.

[0115]FIG. 2 is a block diagram of the actual hardware of the imageprocessing apparatus.

[0116]FIG. 3 is a block diagram showing another application example ofthe image processing apparatus of this invention

[0117]FIG. 4 is a block diagram showing another application example ofthe image processing apparatus of this invention.

[0118]FIG. 5 is a flowchart showing a front stage of main processing inan image processing apparatus according to this invention.

[0119]FIG. 6 is an illustration showing a case where an image variationamount is expressed by component values in orthogonal coordinates.

[0120]FIG. 7 is an illustration showing a case where an image variationamount is expressed by a differential value in adjacent picture elementsin a vertical axis direction and a horizontal axis direction.

[0121]FIG. 8 is an illustration showing a case when an image variationis calculated between adjacent picture elements.

[0122]FIG. 9 is a diagram showing a region wherein a threshold value isvaried.

[0123]FIG. 10 is a flowchart for automatic division of regions.

[0124]FIG. 11 is a diagram showing a region setting.

[0125]FIG. 12 is a diagram showing this region setting in a modifiedexample.

[0126]FIG. 13 is a flowchart showing the latter half of main processing.

[0127]FIG. 14 is a diagram showing the edges obtained by luminancedistribution edge processing and edge processing.

[0128]FIG. 15 is a diagram showing widening of a luminance distributionand a reproducible luminance range.

[0129]FIG. 16 is a diagram showing a conversion table for widening theluminance distribution.

[0130]FIG. 17 is a diagram showing the general concept of brightening byγ correction.

[0131]FIG. 18 is a diagram showing the general concept of darkening by γcorrection.

[0132]FIG. 19 is a diagram showing a correspondence relation forluminance modified by γ correction.

[0133]FIG. 20 is a flowchart showing a case when saturation isemphasized in the latter part of main processing.

[0134]FIG. 21 is a schematic view of a summation state of a saturationdistribution.

[0135]FIG. 22 is a diagram showing a relation between a saturation A andsaturation emphasis index S.

[0136]FIG. 23 is a flowchart showing a case when edges are emphasized inthe latter part of main processing.

[0137]FIG. 24 is a diagram showing the magnitude of image data and astate where image data to be processed is displaced.

[0138]FIG. 25 is a diagram showing a 5×5 picture element unsharp mask.

[0139]FIG. 26 is a block diagram of an image processing system in whichan image processing apparatus according to one embodiment of thisinvention is applied.

[0140]FIG. 27 is a flowchart showing image processing in the imageprocessing apparatus according to this invention.

[0141]FIG. 28 is a diagram showing a sampling frequency.

[0142]FIG. 29 is a diagram showing a picture element sampling number.

[0143] FIGS. 30(a)-(c) are diagrams showing a relation between an imageto be converted and picture elements for sampling.

[0144]FIG. 31 is a diagram showing a block arrangement resulting fromimage division.

[0145]FIG. 32 is a diagram showing a block luminance distribution.

[0146]FIG. 33 is a diagram showing an example of block weighting.

[0147]FIG. 34 is a diagram showing another example of block weighting.

[0148]FIG. 35 is a flowchart showing a case when a feature amount isreevaluated based on an edginess amount.

[0149]FIG. 36 is a diagram showing a relation between a main pictureelement and edge picture elements for determining the edginess amount.

[0150] FIGS. 37(a)-(f) are diagrams showing an example of a filter forcomputing the edginess amount.

[0151]FIG. 38 is a flowchart showing a case when a feature amount isreevaluated based on an edginess amount.

[0152]FIG. 39 is an example of a photographic image.

[0153] FIGS. 40(a)-(d) are diagrams of a luminance distribution of aphotographic image photographed at night.

[0154]FIG. 41 is a block diagram of an image processing system in whichan image processing apparatus according to one embodiment of thisinvention is applied.

[0155]FIG. 42 is a flowchart showing a sampling part in an imageprocessing apparatus according to this invention.

[0156]FIG. 43 is a flowchart showing a feature amount acquiring part andan image processing part.

[0157]FIG. 44 is a block diagram of an image processing system in whichan image processing apparatus according to one embodiment of thisinvention is applied.

[0158]FIG. 45 is a flowchart showing an image processing part in animage processing apparatus according to this invention.

[0159]FIG. 46 is a block diagram showing an image evaluation optioninput screen.

[0160]FIG. 47 is a diagram showing how individual sampling results areconverted to weightings and combined.

[0161]FIG. 48 is a flowchart showing the latter stage of imageevaluation and an image processing part.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0162] Some preferred embodiments of this invention will now bedescribed.

Embodiment 1

[0163] First, a description will be given of one form of image processorcomprising an image processing indicator specifying unit which comprisesan object determining unit, and automatically determines an object aspicture elements having a large variation. Conventionally, thisdetermination had to be performed by a human operator.

[0164]FIG. 1 shows a block diagram of an image processing system towhich an image processing apparatus according to one embodiment of thisinvention is applied. FIG. 2 shows an example of the actual hardwareconstruction by a schematic block diagram.

[0165] In FIG. 1, an image reader 10 outputs photographic image datawhich represented photographs as dot matrix picture elements to an imageprocessing apparatus 20A. The image processing apparatus 20A determinesthe contents and extent of image processing, and then performs theprocessing. The image processing apparatus 20A outputs the processedimage data to the image output apparatus 30, and the image outputapparatus 30 outputs the processed image as dot matrix picture elements.From the image data output by the image processing apparatus 20A, animage variation amount is found for each picture element, and pictureelements having a large variation amount are determined to be those ofthe object. The content and extent of image processing are determined,and image processing is performed in line with this object image data.Therefore, the image processing apparatus 20A comprises an objectdetermining unit which finds an image variation amount in each pictureelement and determines picture elements having a large variation amountas the object, and processing unit which determines the content andextent of image processing in line with object image data.

[0166] A scanner 11 in FIG. 2 and digital still camera 12 or videocamera 14 correspond to a specific example of the image reader 10, thecomputer system corresponds to a specific example of the imageprocessing apparatus 20A comprising a computer 21, hard disk 22,keyboard 23, CD-ROM drive 24, floppy disk drive 25 and modem 26, and theprinter 31 and display 32 correspond to specific examples of the imageoutput apparatus 30. In case of this embodiment, the object is found toperform appropriate image processing, so photographic data such asphotographs are suitable as image data. A modem 26 is connected to thepublic telecommunication line, and to an external network via the publictelecommunication line through which software and data can bedownloaded.

[0167] According to this embodiment, the scanner 11 and digital stillcamera 12 which function as the image reader 10 output RGB (red, green,blue) gradation data. The printer 31 which is the image output apparatus30 requires input of CMY (cyan, magenta, yellow) or CMYK (to which blackis added) as gradation data, and the display 32 requires RGB gradationdata as input. Also, an operating system 21 a runs on the computer 21. Aprinter driver 21 b corresponding to the printer 31 and display driver21 c for the display 32 are built in. Processing is controlled by theoperating system 21 a, and an image processing application 21 d performspredetermined image processing together with the printer driver 21 b anddisplay driver 21 c when necessary. Therefore, the specific role of thiscomputer 21 which functions as the image processing apparatus 20A is toinput RGB gradation data, generate RGB gradation data for optimum imageprocessing, display the data on the display 32 via the display driver 21c, convert the data to CMY (or CMYK) binary data via the printer driver21 b, and print it on the printer 31.

[0168] In this way, according to this embodiment of the invention, acomputer is interposed between the input-output apparatus to performimage processing, but a computer is not absolutely necessary if thesystem is capable of performing various types of image processing onimage data. For example the system may be such that the image processingapparatus which determines the object and performs image processing isbuilt into the digital camera 12 a as shown in FIG. 3. The image isdisplayed on a display 32 a and printed by a printer 31 a usingconverted image data. Alternatively, the object is determined and imageprocessing is performed automatically from image data input via ascanner 11 b and digital still camera 12 b or modem 26 b, as shown inFIG. 4.

[0169] The aforesaid determination of the object and image processingare performed by an image processing program corresponding to aflowchart shown by FIG. 5 which is built into the computer 21. In theflowchart shown in the figure, it is determined whether or not the imageis that of the object.

[0170] According to this invention, picture elements for which the imageis sharp are determined to be those of the object based on theexperimental fact that the image is sharper for the object than forother parts. When image data comprises dot matrix picture elements,gradation data is displayed showing RGB luminance for each pictureelement, and a difference amount between data for adjacent pictureelements becomes large at the edge of the image. This difference amountis a luminance gradient, and is referred to as edginess. In a stepSA110, the edginess of each picture element is determined. When the XYorthogonal coordinate system is considered in FIG. 6, vectors of theimage variation amount may be computed if the X-axis direction componentand Y axis direction component are found respectively. For a digitalimage comprising dot matrix picture elements, assume that there areadjacent picture elements in the vertical axis direction and horizontalaxis direction as shown in FIG. 7, and assume that the luminance isexpressed as f (x, y). In this case f(x, y) is R(x, y), G(x, y), B(x, y)which is the luminance of each of the colors RGB, or it may be a totalluminance Y(x, y). Strictly speaking, the relation between R(x, y), G(x, y), B (x, y) which is the luminance of each of the colors RGB, andtotal luminance Y(x, y), cannot be converted without referring to colorconversion charts, but a simple correspondence relation can be utilizedas described hereafter. As shown in FIG. 7, a difference amount value fxin the X direction and a difference amount value fy in the Y directionmay be written:

fx=f(x+1, y)−f(x, y)   (1)

fy=f(x, y+1)−f(x, y)   (2)

[0171] Therefore the magnitude of the vector |g (X, y)| having thesedifferences as components may be written as:

|g(x,y)|=(fx**2+fy**2)**(1/2)   (3)

[0172] Edginess is of course represented by |g(x, y)|. The pictureelements are actually arranged in length and breadth as a grid shown inFIG. 8, there being eight picture elements in the center. Therefore,expressing a difference amount of image data between adjacent pictureelements as a vector, the sum of this vector may be taken as the imagevariation amount.

[0173] As the edginess may be found for each picture element in thisway, picture elements having a large edginess when compared with athreshold value may be determined to be object picture elements. Howeverfrom actual experience, the object is often situated in the center ofthe figure. This fact provides proof that the use of an arrangementwhere a large number of picture elements are sampled from the centerarea for image processing, gives satisfactory results.

[0174] For this reason, the threshold values Th1, Th2, Th3 used forcomparison in each part of the center of the image are arranged to bedifferent as shown in FIG. 9. In this example, of course, the relation

Th1<Th2<Th3   (4)

[0175] holds. The threshold value is lower the nearer the center, andthis area is determined to be the object even if the edginess isrelatively low.

[0176] As the threshold value varies as shown in the figure, the area isdivided uniformly into three equal parts in the horizontal and verticaldirections from the center of the image. In a step SA120, the thresholdvalue used for comparison is determined based on the area in which thepicture element used for edginess determination is located. Thethreshold value is compared with the edginess in a step SA130, and it isdetermined whether or not the variation amount is large. As a result ofthis comparison, if the edginess is large, it is determined that thispicture element is a picture element of the object, and image data ofthe picture element is stored in a work area in a step SA140. The workarea may be a RAM in the computer 21, or the hard disk 22.

[0177] The above processing is performed for each picture element ofimage data. In a step SA150, the picture element to be processed isdisplaced, and processing is repeated until it is determined thatprocessing has been completed in a step SA160.

[0178] In the embodiment described above, the area division formodifying the threshold value was always based on the central part ofthe image, but the area division may also be varied based on theedginess distribution. FIG. 10 is a flowchart for suitably varying thearea division, and FIG. 11 shows the areas so divided.

[0179] In this case also, subsequent processing is performed on eachpicture element while moving the picture element to be processed in thesame way as above. After the aforesaid edginess was determined in a stepSA210, it is summed in the horizontal axis direction in a step SA220,and summed in the vertical axis direction in a step SA230. Pictureelements to be processed are displaced in a step SA240, and the processloops until it is determined in a step SA250 that processing for allpicture elements is complete.

[0180] After summation is completed for the horizontal axis directionand vertical axis direction, a maximum distribution position on thehorizontal axis is determined in a step SA260, and a maximumdistribution position on the vertical axis is determined in a stepSA270. As shown in FIG. 11, a high edginess part along the horizontaland vertical axis direction is regarded as the center of the image, thearea being divided as follows.

[0181] The distance to an end from the center is divided in half in thevertical direction and the horizontal direction. The threshold value isTh1 for the inside, the remaining distance being divided in half with athreshold value Th2 for the inner area and a threshold value Th3 for theouter area. In a step SA280, by dividing the area as described above,comparison criteria are determined, and in a step SA290, object pictureelements are determined by perform sampling based on edginess by thesame processing as that of the steps SA110-SA160 mentioned aboveaccording to the correspondence between this area and threshold value.

[0182] In this example, after finding the center area, the area wasdivided into two equal parts in the directions of both the horizontalaxis and vertical axis, but the area may also be divided in other ways,e.g. based on the edginess distribution. The actual area division may besuitably varied.

[0183] For example, in the example stated above, summation in thehorizontal axis direction and vertical axis direction were performed inpicture element units, but the image may be divided into a relativelylarger grid as shown in FIG. 12. The summation may then be made in thesegrid units, the position of the maximum distribution determined, andarea division performed.

[0184] If object picture elements can be sampled in this way, theoptimum image processing can be determined and performed based on imagedata for these picture elements. FIG. 13 is a flowchart showing increaseof contrast and lightness compensation as an example.

[0185] In the basic technique to increase contrast according to thisembodiment, a luminance distribution is found based on object imagedata, and if this luminance distribution uses only part of the originalgradation (255 gradations), the distribution is expanded.

[0186] Therefore, a histogram of luminance distribution is generated ina step SA310, and an expansion width is determined in a step SA320. Whenthe expansion width is determined, both ends of the luminancedistribution are found. A luminance distribution of a photographic imageis generally like a hump as shown in FIG. 14. The distribution may ofcourse have various positions and shapes. The width of the luminancedistribution is determined by where the two ends are located, but thepoints where the distribution number is “0” where the distributionslopes away cannot be taken as the ends. There is a case where thedistribution number varies in the vicinity of “O” in the lower slopingpart. This is because from a statistical viewpoint, it changes withoutlimit as it approaches “0”.

[0187] Therefore, the two ends of the distribution are taken to be apart in the distribution shifted somewhere towards the inside by acertain distribution fraction from the brightest side and the leastbright side. In the area in this embodiment, this distribution fractionis set to 0.5%, but the fraction may be modified as deemed appropriate.In this way, white spots and black spots due to noise can also beignored by cutting the upper and lower ends by a certain distributionfraction. Specifically, if such processing is not performed and thereare white spots or black spots, these become the two ends of theluminance distribution. In a 255 gradation luminance value distribution,the lower end is usually “0” and the upper limit is “255”, but the aboveproblem is avoided by considering the end of the distribution to be apoint situated at 0.5% from the end in terms of picture elements.

[0188] In the actual processing, 0.5% of the number of picture elementssampled as the object is computed, the distribution numbers are summedtowards the inside in sequence from the luminance value at the upper endand the luminance value at the lower end in a reproducible luminancedistribution, and the luminance value corresponding to 0.5% isdetermined? Hereafter, the upper limit will be referred to as ymax andthe lower limit as ymin.

[0189] When the reproducible range of luminance is “0”-“255”, theluminance Y converted from the luminance y before conversion, and themaximum value ymax and minimum value ymin of the luminance distribution,is given by the following equations:

Y=ay+b   (5)

where a=255/(ymax−ymin)   (6)

b=−a.ymin or 255−a.ymax   (7)

[0190] In the above equation, when Y<0, Y is set equal to 0, and whenY>255, Y is set equal to 255. a is a slope and b is an offset. Accordingto this conversion equation, a luminance distribution having a certainnarrow width can be enlarged to a reproducible range, as shown in FIG.15. However, when the reproducible range was increased to the maximum toexpand the luminance distribution, highlighted areas are white and highshadow areas come out black. To prevent this, according to thisembodiment, the reproducible range is limited, i.e. a luminance value of“5” is left as a range which is not expanded at the upper and lower endsof the reproducible range. As a result, the parameters of the conversionequation are given by the following equations:

A=245/(ymax−ymin)   (8)

B=5−a/ymin or 250−a/ymax   (9)

[0191] In this case, in the ranges y<ymin and y>ymax, conversion is notperformed.

[0192] However, if this expansion factor (corresponding to a) isapplied, a very large expansion factor may be obtained. For example atdusk, although the width of contrast from the brightest to the darkestpart is naturally narrow, if the contrast of such an image wereconsiderably increased, it would appear to be converted to a daytimeimage. As such a conversion is not desired, a limit is imposed on theincrease factor so that it is equal to or greater than 1.5 (−2). Due tothis, dusk correctly appears as dusk. In this case, processing isperformed so that the center position of the luminance distribution doesnot vary.

[0193] However, in luminance conversion, it is unreasonable to performthe aforesaid conversion (Y=ay+b) on every occasion. This is because theluminance y can only lie within the range “0” to “255”, and theluminance Y after conversion can be found for all possible values of ybeforehand. These can therefore be stored in a table such as is shown inFIG. 16.

[0194] This conversion table corresponds to the expansion widthdetermination processing of the step SA320, and it allows image data tobe converted. However, as it is very useful not only to emphasizecontrast by increasing the luminance range but also to adjust luminanceat the same time, the luminance of the image is determined in a stepSA330 and a correction parameter is generated.

[0195] For example, the hump of the luminance distribution may bedisplaced to the side which is brighter overall as shown by the dottedline in FIG. 17 when the hump is nearer dark as shown by the solid linein the figure. Conversely, the hump of the luminance distribution may bedisplaced to the side which is darker overall as shown by the dottedline in FIG. 18 when the hump is nearer bright on the whole as shown bythe solid line in the figure.

[0196] By performing various experiments, according to this embodiment,a Median ymed in the luminance distribution is found. When the medianymed is less than “85”, the image is determined to be dark, and islightened by a γ correction corresponding to the following γ value:

γ=ymed/85   (10)

or

γ=(ymed/85)**(1/2)   (11).

[0197] In this case even if γ<0.7, γ is set equal to 0.7. If such alimit is not provided, a night scene appears as if it is in daylight. Ifthe image is made too light, it becomes too white and contrast tends tobe too low, hence it is preferable to perform processing such asemphasis in conjunction with saturation.

[0198] On the other hand, when the median ymed is greater than “128”,the image is determined to be a light image and is darkened by a γcorrection corresponding to the following γ value:

γ=ymed/128   (12) or

γ=(ymed/128)**(1/2)   (13).

[0199] In this case even if γ>1.3, γ is set equal to 1.3 so that theimage does not become too dark.

[0200] This γ correction may be applied to the luminance distributionbefore conversion, or to the luminance distribution after conversion.Correspondence relations for γ correction are shown in FIG. 19. Whenγ<1, the curve bulges upwards, whereas when γ>1, the curve bulgesdownwards. Of course, the result of this γ may also be reflected in thetable shown in FIG. 16, and the same correction may be applied to tabledata.

[0201] Finally, in a step SA340, it is determined whether or notcontrast correction and lightness compensation are necessary. In thisdetermination, the aforesaid expansion factor (a) and γ value arecompared with suitable threshold values, and when the expansion factoris large and the γ value exceeds a predetermined range, it is determinedthat such correction is necessary. If it is determined to be necessary,conversion of image data is performed. Specifically, the need for imageprocessing and its extent are assessed in the steps SA310-SA340, and therequired image processing is performed in the a SA350. The processingunit to accomplish this comprises hardware and software.

[0202] Conversion on the basis of equation (5) is performed when it isdetermined that image processing is necessary. This equation may also beapplied to correspondence relations between RGB component values. Thecomponent values after conversion (R, G, B) relative to the componentvalues before conversion (R0, G0, B0) may be found from:

R=a·R 0 +b   (14)

G=a·G 0 +b   (15)

B=a·B 0 +b   (16)

[0203] Herein, the RGB component values (R0, G0, B0), (R, G, B) have thesame range when the luminance y, Y has the gradation“0” “255”, so theaforesaid conversion table of luminance y, Y may be used withoutmodification.

[0204] Therefore, the conversion table corresponding to equations(14)-(16) is referred to for the image data (R0, G0, B0) for all pictureelements in a step SA350, and the process for obtaining the image data(R, G, B) after conversion is repeated.

[0205] In this processing unit, the determination is performed only forcontrast correction and lightness compensation, but specific examples ofimage processing are not limited to this.

[0206]FIG. 20 shows a flowchart for performing image processing forsaturation emphasis.

[0207] First, if the object and determined picture element data havesaturation as a component element, a distribution may be found usingsaturation values. However, as the data comprises only RGB componentvalues, saturation values cannot be obtained unless they are convertedto a color specification space which comprises direct component values?For example, in a Luv space which is a colorimetric system, the L axisrepresents luminance (lightness), and hues are represented by the U axisand V axis. Herein, as the distance from the intersection point of the Uaxis and V axis shows saturation, the saturation is effectively(U**2+V**2)**(1/2).

[0208] Such a color conversion between different color specificationspaces requires an interpolation to be performed while referring to acolor conversion table which stores correspondence relationships, andthe computation amount is enormous. In view of this, according to thisembodiment, standard RGB gradation data is utilized directly as imagedata, and saturation substitute values X are found as follows:

X=|G+B−2×R|  (17)

[0209] Actually, the saturation is “0” when R=G=B, and is a maximumvalue either for any of the single colors RGB or for a mixture of twocolors in a predetermined proportion. Due to this, it is possible toappropriately represent saturation directly. For yellow which is amixture of the color red with green and blue, from the simple equation(17), the maximum saturation value is obtained, and when the componentsare equal, this is “0”. For green or blue alone, about half the maximumvalue is attained. Of course, substitutions may be made using theequations:

X′=|R+B−2×G|  (18)

X″=|G+R−2×B|  (19)

[0210] In a step SA410, a histogram distribution is found for thesaturation substitution value X. In equation (17), saturation isdistributed in the range of minimum value “0”-maximum “511”, and thedistribution obtained is approximately as shown in FIG. 21. In a nextstep SA420, based on the summed saturation distribution, a saturationindex is determined for this image. According to this embodiment, arange occupied by the upper 16% of distribution number is found withinthe number of picture elements determined to be the object. Assumingthat the lowest saturation “A” in this range represents the saturationof the image, a saturation emphasis index S is determined based on thefollowing equation. In other words, it is assumed that:

If A<92, S=−A×(10/92)+50   (20)

If 92≦A<184, S=−A×(10/46)+60   (21)

If 184≦A<230, S=−A×(10/23)+100   (22)

If 230≦A, S=0   (23)

[0211]FIG. 22 shows a relation between this saturation “A” and thesaturation emphasis index S. As shown in the figure, in the rangebetween the maximum value “50”-minimum value “0”, the saturation index Sgradually varies so that it is large when the saturation A is small, andsmall when the saturation A is large.

[0212] When saturation is emphasized based on the saturation emphasisindex S, if the image data is provided with saturation parameters asstated above, the parameters may be converted. When an RGB colorspecification space is adopted, the data must first be converted intothe Luv system which is a standard color system, and moved in a radialdirection within the Luv space. However, this means that RGB image datamust first be converted into image data in Luv space, and then returnedto RGB after saturation emphasis which involved an enormous amount ofcomputation. Therefore, RGB gradation data are used without modificationfor saturation emphasis.

[0213] When the components are component values of hue components whichare in a schematic pair relation as in the case of the RGB colorspecification space, the color is grey and there is no saturation ifR=G=B. Therefore if the component which has the minimum value in RGB isconsidered merely to have a reduced saturation without: having anyeffect on the hue of the picture elements, the minimum of each componentmay be subtracted from all the component values, and the saturationemphasized by increasing the value of the difference.

[0214] First, a parameter Sratio which is useful for calculation isfound from the aforesaid saturation emphasis index S by the equation:

Sratio=(S+100)/100   (24)

[0215] In this case the saturation emphasis parameter Sratio=1 when thesaturation emphasis index S=O, and saturation is not emphasized. Next,assuming that the value of the blue (B) component in the components (R,G, B) of the RGB gradation data is the minimum, this saturation emphasisparameter Sratio is used to perform the following conversion:

R′=B+(R−B)×Sratio   (25)

G′=B+(G−B)×Sratio   (26)

B′=B   (27)

[0216] As a result, two-way color conversions between the RGB colorspecification space and the Luv space are rendered unnecessary, andcomputing time can be reduced. In this embodiment, as for non-saturationcomponents, the component with the minimum value was simply subtractedfrom other component values, but other conversion equations may be usedto subtract non-saturation components. However when only the minimumvalues are subtracted as in equations (25)-(27), there are nomultiplications or divisions so the computation is easier.

[0217] When the equations (25)-(27) are used, a good conversion ispossible, however in this case when saturation is emphasized, luminancealso increases so the image becomes lighter overall. Therefore, theconversion is performed using a difference value obtained by subtractingan equivalent luminance value from each component value.

[0218] Firstly to find the luminance, as the computation becomes bulkywhen a color conversion is performed in Luv space, the followingequation which is used in television for example, is used to find theluminance from RGB. The luminance Y is given by

Y=0.30 R+0.59G+0.11B   (28).

[0219] It will be assumed that saturation emphasis is given by

R′=R+ΔR   (29)

G′=G+ΔG   (30)

B′=B+ΔB   (31).

[0220] These addition/subtraction values ΔR, ΔG, ΔB are found by thefollowing equations based on difference amount values:

ΔR=(R−Y)×Sratio   (32)

ΔG=(G−Y)×Sratio   (33)

ΔB=(B−Y)×Sratio   (34)

[0221] As a result, the conversion can be performed by

R′=R+(R−Y)×Sratio   (35)

G′=G+(G−Y)×Sratio   (36)

B′=B+(B−Y)×Sratio   (37)

[0222] Conservation of luminance is clear from the following equations:

Y′=Y+ΔY   (38) $\begin{matrix}\begin{matrix}{{\Delta \quad Y} = {{0.30\quad \Delta \quad R} + {0.59\Delta \quad G} + {0.11\Delta \quad B}}} \\{= {{Sratio}\left\{ {\left( {{0.30R} + {0.59G} + {0.11B}} \right) - Y} \right\}}} \\{= O}\end{matrix} & (39)\end{matrix}$

[0223] Also, when the input is grey (R=G=B), the luminance Y=R=G=B, theaddition/subtraction values ΔR=ΔG=ΔB=0 and there is no color in theachromaticity. If equations (35)-(37) are utilized, the luminance isstored, and the image does not become lighter overall even if saturationis emphasized.

[0224] If the saturation emphasis index Sratio is found as describedabove, it is compared with a predetermined threshold value in a stepSA430, and it is determined whether the image requires saturationemphasis. If it is necessary, the image data for all picture elements isthen converted based on equations (35)-(37) in a step SA440.

[0225] Therefore, in the steps SA410-SA430, the need for saturationemphasis processing and its extent is determined, and when it is deemednecessary in the step SA430, saturation emphasis processing isperformed. A processing unit therefore comprises hardware and softwareto accomplish these functions.

[0226] To determine the content and extent of image processing based onobject picture elements, edge emphasis processing may also be used. FIG.23 shows a flowchart of this edge emphasis processing. As object pictureelements are selected, the edginess is averaged for object pictureelements by dividing the integrated edginess by the number of pictureelements in a step SA510. If the number of picture elements is E(I) Pix,the sharpness degree SL of the object image may be computed by$\begin{matrix}{{SL} = {\sum\limits_{x,y}{{{{g\left( {x,y} \right)}}/{E(I)}}\quad {{pix}.}}}} & (40)\end{matrix}$

[0227] In this case, the degree of sharpness is lower (the image appearsmore blurred) the lower the SL value of the image, and the degree ofsharpness is higher (the image appears clearer) the higher the SL valueof the image.

[0228] On the other hand because the sharpness of the image issubjective, the degree of sharpness SL is found in the same way forimage data having an optimum sharpness obtained experimentally, thisvalue is set as an ideal sharpness SLopt, and a degree of edge emphasisEenhance is found from the relation

Eenhance=ks·(SLopt−SL)**(1/2)   (41)

[0229] in a step SA520.

[0230] Herein, ks varies based on the magnitude of the image, and whenthe image data comprises height dots and width dots in the vertical andhorizontal directions as shown in FIG. 24, ks is found from

ks=min(height,width)/A   (42)

[0231] Herein, min (height, width) denotes the smaller of height andwidth, and A is the constant “768”. It will be understood that theserelations are obtained from experiment, and may be modified asappropriate. However, good results are basically obtained by making thedegree of emphasis larger the larger the image.

[0232] When the degree of edge emphasis Eenhance is found in thismanner, it is compared with a predetermined threshold value in a stepSA530, and it is determined whether edge emphasis is necessary. If it isdeemed to be necessary, edge emphasis processing is performed on allpicture elements in a step SA540.

[0233] In edge emphasis processing, a luminance Y′ after emphasisrelative to the luminance Y of each picture element before emphasis iscomputed by:

Y′=Y+Eenhance·(Y−Yunsharp)   (43)

[0234] Herein, Yunsharp is unsharp mask processing relative to imagedata of each picture element. Unsharp mask processing will now bedescribed. FIG. 25 shows an example of a 5×5 picture element unsharpmask. In this unsharp mask 41, the central value “100” is a weighting ofa picture element Y (x,y) in matrix image data, which is used formultiplication with weighting corresponding to a numerical value in thegrid of the mask for edge picture elements. When this unsharp mask 41 isutilized, multiplication is performed based on the computationalequation $\begin{matrix}{{{Yunsharp}\quad \left( {x,y} \right)} = {\left( {1/396} \right){\sum\limits_{i,j}\left( {{Mij} \times {Y\left( {{x + 1},{y + j}} \right)}} \right)}}} & (44)\end{matrix}$

[0235] In equation (44), “396” is a total value of weightingcoefficients, and it is the total value of each grid division in unsharpmasks of different size. Mij is a weighting coefficient written in agrid division of the unsharp mask, and Y (x, y) is image data for eachpicture element. ij is expressed in horizontal and vertical coordinatevalues for the unsharp mask 41.

[0236] The meaning of the edge emphasis computation based on equation(43) is as follows. As Yunsharp (x, y) is added by making the weightingof edge picture elements lower than that of main picture elements, theresult is “unsharp” image data. Images which are made unsharp in thisway have the same meaning as those subjected to a low pass filter.Therefore, “Y(x, y)−Unsharp(x,y)” means low frequency components areremoved from the total components, which has the same meaning asapplying a high pass filter. If high frequency components which passedthrough the high pass filter are multiplied by the edge emphasisEenhance and added to “Y(x, y)”, the high frequency components areincreased in direct proportion to the edge emphasis Eenhance, and theedges are thereby emphasized. Considering the situation when edgeemphasis is needed, edge emphasis refers to the edge of the image, andit therefore may be computed only when there is a large difference ofimage data between adjacent picture elements. If this is done, there isno need to compute an unsharp mask for most image data which is not edgeparts, so the amount of processing is vastly reduced.

[0237] In the actual computation, if we write:

Δ=Y−Y′  (45)

[0238] from the luminance Y′ after emphasis and the luminance Y beforeemphasis, R′ G′ B′ after conversion may be computed as:

R′=R+delta

G′=G+delta

B′=B+delta   (46).

[0239] Therefore, in this edge emphasis processing in the stepsSA510-SA530, the need for edge emphasis and its extent are determined,and image processing is performed when it is determined to be necessaryin the step SA530. A processing unit comprising hardware and software isprovided to perform these functions.

[0240] It is determined whether to perform image processing regardingcontrast correction, lightness compensation, saturation emphasis andedge emphasis. However, it is not absolutely necessary to make a choiceas to whether or not to perform image processing. Specifically, anemphasis degree may be set for each, and image processing performed withthe set emphasis degree. Of course, in this case also, the contents andextent of image processing which should be performed are determined, andthe processing is performed.

[0241] Next, the operation of this embodiment having the aforesaidconstruction will be described.

[0242] A photographic image is read by the scanner 11, and printed bythe printer 31. Specifically, when the operating system 21 a is runningon the computer 21, the image processing application 21 d is started,and reading of the photograph is started by the scanner 11. After theread image data has been assimilated by the image processing application21 d via the operating system 21 a, the picture elements to be processedare set in initial positions. Next, the edginess is determined based onequations (1)-(3) in the step SA110, and the edginess is compared with athreshold value in the step SA120. When the edginess is large, it isdetermined that the picture element to be processed is an edge pictureelement, and image data for the corresponding picture element is storedin the work area in the step SA130. In the step SA140, it is determinedwhether or not picture elements to be processed are to sampleduniformly, and if so, in the step SA150, image data for these pictureelements is stored in the work area. The above processing is repeateduntil it is determined to have been performed for all picture elementsin the step SA160 while picture elements to be processed are displacedin the step SA150.

[0243] When this has been performed for all picture elements, image datafor picture elements determined to be those of the object are stored inthe work area. Therefore, even if the situation of a photographic imageread from the image data in this work area is determined, the nature ofthe image is not misinterpreted due to the effect of the background,etc. According to this embodiment, image data was stored in the workarea, but from the viewpoints of memory capacity and processing time, itis not absolutely necessary to store image data itself in the work area.Specifically, histograms of luminance distribution or saturationsubstitute value distribution for picture elements determined to bethose of the object are generated, and histogram information maytherefore be pre-stored in the step SA140.

[0244] When contrast correction and luminance compensation are performedautomatically, a histogram of luminance distribution is found in thestep SA140 or the step SA310, a parameter for expansion processing isdetermined based on equations (8), (9) in the step SA320, and aparameter for luminance compensation is determined based on equations(10)-(13) in the step SA330. These parameters are compared with apredetermined threshold value in the step SA340, and if it is determinedthat image processing should be performed, the luminance is convertedbased on the aforesaid parameter in the step SA350. In this case, theluminance conversion table shown in FIG. 16 may be generated to reducethe computing amount, and image data may be converted based on equations(14)-(16).

[0245] Subsequently, the processed image data may be displayed on thedisplay 32 via the display driver 21 c, and if it is satisfactory, it isprinted by the printer 31 via the printer driver 21 b. In other wordsthe printer driver 21 b inputs RGB gradation data with emphasized edges,performs rasterization corresponding to the print head area of theprinter 31 after a predetermined mapping, color converts rasterized datafrom RGB to CMYK, converts CMYK gradation data to binary data, andoutputs it to the printer 31.

[0246] Due to the above processing, photographic image data read via thescanner 11 is automatically subjected to the optimum contrast correctionand lightness compensation, displayed by the display 32, and printed bythe printer 31. In other words, it is determined whether or not contrastcorrection and lightness compensation are necessary based on the objectpart of the corresponding photographic image, and if it is necessary,the optimum degree of image processing is performed. When object pictureelements are sampled, although edginess is required to determineparameters, it is not necessary to find the edginess for all pictureelements. The edginess may be determined for sampled picture elements,and a determination made as to whether or not the picture elements arethose of the object.

[0247] The invention is not limited to this contrast correction andluminance compensation. In the case also of saturation emphasis and edgeemphasis, picture elements for which there is a large variation amountare determined to be those of the object. The content and extent ofimage processing are determined based on image data for object pictureelements, and the necessary processing is performed.

[0248] Hence, the computer 21 which is the core of image processingcalculates the edginess, which is the image variation amount, from datafor adjacent picture elements in the step SA110, selects only imageswhich have a large edginess and determines them to be object pictureelements in the steps SA120, SA130, and calculates optimum parametersfor performing contrast correction and lightness compensation from imagedata for object picture elements in the steps SA310-SA330. An imageprocessing indicator can therefore be determined based on image data forobject picture elements, and optimum image processing can be performedautomatically.

Embodiment 2

[0249] Next, an embodiment of an image processing apparatus will bedescribed wherein an image processing indicator specifying unitcomprises a feature amount uniform sampling unit, and a predeterminedweighting is applied after sampling without a large computational amountin the sampling stage so as to automatically perform optimum imageprocessing.

[0250]FIG. 26 is a block diagram of an image processing system to whichthe image processing apparatus according to one embodiment of thisinvention is applied. The actual hardware construction may be similar tothe system shown in FIG. 2.

[0251] In FIG. 26, an image reader 10 outputs photographic image datarepresented as dot matrix picture elements to an image processingapparatus 20B, and the image processing apparatus 20B performs imageprocessing after determining a degree of emphasis via a predeterminedprocess. The image processing apparatus 20B outputs the processed imagedata to an image output device 30, and the image output device 30outputs the processed image in dot matrix picture elements. Herein, theimage data which the image processing apparatus 20B outputs isobtained:by uniformly sampling feature amounts from picture elements bya predetermined criterion, reevaluating them with a predeterminedweighting, and processing them with an emphasis degree determinedaccording to the reevaluated feature amounts. Therefore, the imageprocessing apparatus 20B comprises a feature amount uniform samplingunit which uniformly samples feature amounts, a feature amount weightingreevaluation unit which reevaluates sampled feature amounts with apredetermined weighting, and a processing unit which performs imageprocessing with the degree of emphasis according to the reevaluatedfeature amounts.

[0252] Specifically, the determination of the object and accompanyingimage processing are performed by an image processing program in theaforesaid computer 21 corresponding to a flowchart shown in FIG. 27. Inthe flowchart shown in the figure, image processing is performed toadjust the contrast of the image. After sampling luminance which is afeature amount while uniformly thinning out picture elements from thewhole image in a step SB110, this feature amount is reevaluated byapplying a predetermined weighting in a step SB120, and image processingto adjust luminance is performed in steps SB130-SB160.

[0253] In the step SB110, the luminance of each picture element in dotmatrix image data in the horizontal and vertical directions is found asshown in FIG. 24, and a histogram is generated. In this case, if theprocessing is applied to all picture elements it can be said to beprecise, but as the summation results are reevaluated by weighting, itdoes not necessarily have to be precise. Therefore picture elementswherefrom the luminance has been sampled to within the limits of acertain error are thinned out, processing amount is reduced, and theprocess is thereby speeded up. According to statistical error, an errorfor a sample number N can generally be represented by

[0254] 1/(N**(1/2)) where ** represents involution.

[0255] Therefore in order to perform processing with an error of around1%, N=10000.

[0256] Herein, the bitmap screen shown in FIG. 24 is number of(width)×(height) picture elements, and a sampling period ratio is givenby

ratio=min(width, height)/A+1   (47).

[0257] min (width, height) is the smaller of width and height Herein,and A is a constant. The sampling period ratio mentioned here expresseshow frequently to perform sampling in numbers of picture elements, andthe mark O in FIG. 28 shows the case where the sampling period ratio=2.In other words, one picture element is sampled every two pictureelements in the vertical and horizontal directions, so sampling isperformed every other picture element. The number of sampling pictureelements in 1 line when A=200 is as shown in FIG. 29.

[0258] As is clear from the figure, except for the case when thesampling period ratio=1 when sampling is not performed, at least 100picture elements are sampled when there is a width of 200 pictureelements or more. Therefore, when there are 200 or more picture elementsin the vertical and horizontal directions (100 picture elements)×(100picture elements)=10000 picture elements are sampled and the error is 1%or less.

[0259] The reason for taking min (width, height) as a basis is asfollows. For example, as shown by the bitmap in FIG. 30(a), ifwidth>>height and the sampling period ratio is determined by width whichis the longer direction, only two lines of picture elements, i.e. thetop edge and bottom edge, can be sampled in the vertical direction asshown in (b).

[0260] However, if the sampling period ratio is determined based on thesmaller of the two as min (width, height), thinning which includes themiddle part can be performed even in the lesser, vertical direction asshown in (c). In other words, sampling with a predetermined number ofsamplings can be guaranteed.

[0261] Here, the feature which is sampled with thinning of pictureelements is luminance. As described above, according to this embodiment,the data handled by the computer 21 is RGB gradation data, and it doesnot directly have luminance values. To calculate luminance, a colorconversion to the Luv color specification space could be performed, butthis is not a good solution due to the problem of computing amount.

[0262] For this purpose, the aforesaid equation (28) which calculatesluminance directly from RGB and is used for television, etc., isutilized here.

[0263] Also, the luminance histogram is not summed for the whole image,but the input image is divided into 3 horizontal blocks and 5 verticalblocks, i.e. a total of 15 blocks, and the summation carried out foreach block, as shown in FIG. 31. According to this embodiment there are15 blocks, however there is of course no restriction on the blockdivision used.

[0264] In particular, with printer drivers, etc., image data is receivedfrom an application in block units, and these blocks may also be used todemarcate areas for weighting.

[0265] The reason for summing in blocks in this way is to reduce theprocessing amount required. As reevaluation with weighting is performedin the step SB120 it is not absolutely necessary to perform thesummation for each block, and the summation may instead take the form ofa histogram which considers the weighting for each selected pictureelement. Moreover, as the weighting changes relative to the summationresult regardless of blocks, the summation can also be made with onehistogram using a weighting depending on the block. FIG. 32 is a figureshowing an example of luminance distribution of the block Bi.

[0266] When the summation is to performed for each block, a reevaluationis made by weighting according to area in the step SB120. FIG. 33 andFIG. 34 show examples of weighting each block. In the case of anordinary photographic image, the object being photographed is usually inthe center. In this sense, the feature amount should be evaluatedplacing more on the center part of the image data. On the other hand, inthe case of a souvenir photo taken in front of a building, the personbeing photographed is generally in bottom center. Specifically, theperson is usually in the lower part of the image with respect to heightabove ground. In this case therefore, the feature amount should beevaluated by weighting the lower middle part of the image. FIG. 33 showsan example of the former case, and FIG. 34 shows an example of thelatter.

[0267] If the weighting of each block=Wi(i) (i=1-15) and the weighted,reevaluated luminance distribution is DY, $\begin{matrix}{{SP} = {\sum\limits_{i = {1\sim 15}}{Wi}}} & (48)\end{matrix}$

 Ki =Wi/SP   (49)

[0268] then: $\begin{matrix}{{DY} = {\sum\limits_{i = {1\sim 15}}{{Ki}*{dYi}}}} & (50)\end{matrix}$

[0269] After the histogram of the reevaluated luminance distribution isobtained in this way, the intensity of image processing is calculatedfrom this feature amount. In other words, the width for increasingcontrast is determined. This is done exactly as described above, i.e. amaximum value ymax, minimum value ymin and median ymed are acquired inthe step SB130, the expansion factor a and offset b are found in thestep SB140, and processing to generate a conversion table is performedin the step SB150. In the step SB160, for image data (R0, G0, B0) forall picture elements, the processing to obtain image data (R, G, B)after conversion is repeated while referring to conversion tablescorresponding to equations (14)-(16).

[0270] A processing unit is provided comprising the hardware andsoftware to perform the steps SB130-SB160.

[0271] It will moreover be understood that although according to thisembodiment, the image processing described is that of contrastcorrection and lightness compensation, the invention may be applied inexactly the same way to other image emphasis processing.

[0272] In the above processing, the feature amount was reevaluated byweighting according to a position in the image. However the weightingcriterion is not limited to this, and various other types of criterionare possible. As an example, FIG. 35 shows a flowchart for the casewhere the object being photographed is detected from the image variationamount, and the weighting is varied.

[0273] A step SB210 replaces the above step SB110, and luminance issummed while thinning out picture elements uniformly. However, insteadof summation only of luminance, edginess can also be summed as shownbelow.

[0274] In the input image it can be said that contrast of the backgroundchanges gradually; on the other hand, the object is sharp and so thereis an intense change of luminance. Therefore, a difference of density isfound between one picture element and surrounding picture elements asshown in FIG. 36. This density difference is taken as the edginess ofthe picture element considered. This density difference may be computedby applying a filter. FIG. 37(a)-(f) show several examples of such afilter, and show weighting coefficients when the luminance is weightedfor a specific picture element and eight surrounding picture elements.Here, in the case of (a), the weighting is found for nine pictureelements, so nine multiplications and eight additions are necessary tocalculate the edginess of each picture element. When the image becomeslarge, it is impossible to ignore this computing amount, so fivemultiplications and four additions are performed in (b), (c), threemultiplications and two additions are performed in (d), (e) and oneaddition are performed in (f).

[0275] In these examples, the picture element in question is comparedonly with surrounding picture elements. With the “unsharp mask”, thesharpness of the considered picture element may be found by using awider range of image data. However, as edginess in this embodiment isonly a tool to evaluate the weighting per block, even the filters ofthese examples of reduced computation amount give sufficiently goodresults.

[0276] The summation of edginess may be performed by summing theedginess of picture elements for each block. Alternatively, when theabsolute value of this edginess is larger than a predetermined thresholdvalue absolutely, the picture element is determined to be an edgepicture element, and the total number of edge picture elements for eachblock is summed. When the edginess in each block is written as ERi(i=1-15), this total number SE is given by: $\begin{matrix}{{SE} = {\sum\limits_{i = {1\sim 15}}{ERi}}} & (51)\end{matrix}$

[0277] so the weighting coefficient KEi itself may be expressed as

KEi=ERi/SE   (52).

[0278] Therefore the luminance distribution DY which is reevaluated byweighting may be calculated as: $\begin{matrix}{{DY} = {\sum\limits_{i = {1\sim 15}}{{KEi}*{dYi}}}} & (53)\end{matrix}$

[0279] Also, if the total number of edge picture elements in each blockis ENi (i=1-15), the total number SE is: $\begin{matrix}{{SE} = {\sum\limits_{{i = 1},15}{{ENi}.}}} & (54)\end{matrix}$

[0280] The weighting coefficient KEi itself is expressed as

KEi=ENi/SE   (55),

[0281] so the luminance distribution DY which was reevaluated byequation (53) can be obtained. In any case, the luminance distributionDY is reevaluated based on the computational equation (53) in the stepSB220.

[0282] In this example, the luminance of picture elements determined tobe edge picture elements is not sampled. Instead, the edginess and totalnumber of edge picture elements is merely used for determining blockweighting coefficients. In other words, instead of summing the featureamounts for picture elements having a specific property (edginess), anaverage feature amount which is not uneven can be obtained for theblock.

[0283] If the luminance distribution is reevaluated in this way, thecontrast may be increased and the lightness may be modified by theprocessing of the aforesaid steps SB130-SB160.

[0284] In view of the fact that the original object is often that of aperson, a reevaluation may also be made by placing more weight onpicture elements with skin color. FIG. 38 shows a flowchart whereinattention is paid to a specific color to determine block weightingcoefficients.

[0285] In a step SB310 corresponding to the step SB110, luminance issummed by the same thinning process, it is determined whether or notpicture elements appear to have skin color based on the chromaticity ofeach picture element, and all picture elements having skin color arethen summed. For chromaticity, x-y chromaticity is calculated for eachpicture element. Now, if

r=R/(R+G+B)   (56)

g=G/(R+G+B)   (57)

[0286] when the RGB gradation data of object picture elements in the RGBcolorimetric system is (R, G, B), the following relationships existbetween chromaticity coordinates x, y in the X, Y, Z colorimetricsystem:

x=(1.1302+1.6387r+0.6215g)/(6.7846−3.0157r−0.38579)   (58)

y=(0.0601+0.9399r+4.5306g)/(6.7846−3.0157r−0.3857g)   (59).

[0287] Herein, as chromaticity represents an absolute proportion of acolor stimulation value without it being affected by lightness, it maybe said that the nature of an object can be determined from thechromaticity of its picture elements. Since

0.35<x<0.40   (60)

0.33<y<0.36   (61)

[0288] in the case of skin color, it may be considered that if a pictureelement is within this range when chromaticity is determined for eachpicture element, that picture elements shows a person's skin, and thenumber of skin color picture elements in the block is increased by one.

[0289] After the number of skin color picture elements is obtained inthis way, in the next step SB320, weighting coefficients are determinedas in the case of edge picture elements described above, and theluminance distribution DY is reevaluated. Specifically, if the number ofskin color picture elements in each block is written as CNi (i=1-15),the total number SC of such elements is: $\begin{matrix}{{SC} = {\sum\limits_{i = {1\sim 15}}{CNi}}} & (62)\end{matrix}$

[0290] Therefore the weighting coefficient KCi is expressed by

KCi=CNi/SC   (63),

[0291] and the luminance distribution DY reevaluated with this weightingmay be calculated by $\begin{matrix}{{DY} = {\sum\limits_{i = {1\sim 15}}{{KCi}*{{dYi}.}}}} & (64)\end{matrix}$

[0292] Also in this example, the luminance of picture elementsdetermined to be skin color picture elements is not sampled, the totalnumber of skin color picture elements merely being used for determiningblock weighting coefficients. Therefore an average feature amount thatis not uneven can be obtained for the block. In this case too, after theluminance distribution has been reevaluated in this way, the contrastmay be corrected and lightness compensated by the processing of theaforesaid steps SB130-SB160. In the case of the photograph shown in FIG.39, a girl appears in the center, and picture elements of the face, armsand legs are determined to be skin color picture elements. Of course,the chromaticity may also be found and picture elements summed for othercolors.

[0293] Until now, the weighting coefficient was determined by onefactor, but the importance of each factor may also be added and theabove processing applied repeatedly. When the weighting coefficient ofeach block Bi (i=1-15) is Tji for a factor j (1=position in image,2=edginess, 3=skin color picture element number), the weighting Tjidistributed among blocks for each factor is a temporary weighting,$\begin{matrix}{{Sj} = {\sum\limits_{i = {1 \sim 15}}^{\quad}\quad {Tji}}} & (65)\end{matrix}$

 and

Kji=Tji/Sj   (66),

[0294] then the real weighting coefficient Ki in the block Bi is givenby $\begin{matrix}{{Ki} = {\sum\limits_{j = {1 \sim 3}}^{\quad}\quad {{Aj}*{{Kji}.}}}} & (67)\end{matrix}$

[0295] Aj is a coefficient to represent the importance of each factor,and is suitably determined so that the total number is 1. If skin coloris emphasized as an example, the settings A1=0.2, A2=0.2, A3=0.6 arepossible.

[0296] Next, the effect of this embodiment having the aforesaidconstruction will be described. First, a description will be given alongthe lines of the previous embodiment.

[0297] A photographic image is read by the scanner 11, and printed bythe printer 31. Specifically, when the operating system 21 a is runningon the computer 21, the image processing application 21 d is started,and reading of the photograph is started by the scanner 11. After theread image data has been assimilated by the image processing application21 d, the luminance of picture elements is summed while thinning in thestep SB110. The summed luminance distribution dYi is reevaluated in thestep SB120 based on determined weightings corresponding to the positionof each block shown in FIG. 33 and FIG. 34, and ymax, ymin and ymed arecalculated in the step SB130 based on the reevaluated distribution DY.

[0298] In the next step SB140, the slope a and offset b which areemphasis parameters are computed based on equations (8) or (9), the γvalue of the γ correction required for lightness compensation iscalculated based on equations (10)-(13), and the conversion data shownin FIG. 16 is generated in the step SB150. Finally, in the step SB160,the image data for all picture elements is converted by referring tothis conversion table.

[0299] Using the weightings shown in FIG. 33, the weighting is higherfor blocks near the center, so the summed luminance distribution is alsoweighted more heavily the closer it is to the center. Assume for examplethat a person is photographed at night using flash. Even if a goodoverall luminance distribution is obtained for the person as shown inFIG. 40(a) due to the effect of the flash, the part surrounding theperson is dark, and here the luminance distribution is shifted towardsthe dark side as shown in (b). In this case if an average were merelytaken, a luminance distribution shifted towards the dark side overallwould still be obtained as shown in (c), and if the contrast werecorrected and the lightness compensated, the image would only be madetoo light overall and would not be satisfactory.

[0300] However, if more weighting is given to the center block as shownin FIG. 33, a luminance distribution DY is obtained which is stronglyaffected by the luminance distribution in the center of the image asshown in FIG. 40(d). Hence, the intensity of image processing based onthis distribution is no longer a matter of over-emphasizing contrast andover-compensating lightness.

[0301] Conversely, if a person is photographed with a backlight, theface will be dark, and the background will be light, so a good luminancedistribution will not necessarily be obtained overall. Yet even in thiscase, by giving more weight to the luminance distribution of centerblocks where the face is dark as shown in FIG. 33, the dark luminancedistribution is reflected, and image processing is performed to increasecontrast and compensate lightness.

[0302] Due to the aforesaid processing, photographic image data read viathe scanner 11 is processed automatically with optimum intensity,displayed on the display 32, and printed by the printer 31.

[0303] The computer 21 which is the core of image processing sums theluminance distribution which a feature amount for each area whileuniformly selecting picture elements in the step SB110. In the stepSB120, a reevaluation is performed with weightings determined for eacharea, and a luminance distribution strongly influenced by the intrinsicluminance distribution of the object can thus be obtained while uniformsampling is performed. The intensity of image processing is determinedbased on this luminance distribution in the steps SB130-SB150, and theimage data is converted in the step SB160. Hence, image processing isperformed with optimum intensity while the amount of processing isreduced.

Embodiment 3

[0304] Next, an embodiment will be described wherein the aforesaid imageprocessing indicator specifying unit comprises an evaluation unit thatobtains feature amounts based on a plurality of predetermined criteria,and the aforesaid processing unit which can convert image data by aplurality of methods, uses feature amounts according to the method.

[0305]FIG. 41 shows a block diagram of an image processing system whichapplies the image processing apparatus according to one embodiment ofthis invention. A typical hardware construction is similar to the systemshown in FIG. 2.

[0306] In FIG. 41, the image reader 10 outputs photographic image dataas dot matrix picture elements to the image processing apparatus 20C.When plural image processings are applied, the image processingapparatus 20C determines plural feature amounts based on the optimumcriterion for each type of processing, and carries out each type ofprocessing using the most appropriate feature amount. The imageprocessing apparatus 20C outputs the processed image data to the imageoutput device 30, and the image output device outputs the processedimage in dot matrix picture elements.

[0307] The image processing apparatus 20C first sums the image dataaccording to plural evaluation criteria, and thereby obtains pluralfeature amounts. In this sense, the image processing apparatus 20Ccomprises an evaluation unit, and as it performs image processing basedon feature amounts which have been selected according to the imageprocessing content, it may be said to further comprise an imageprocessing unit.

[0308] The acquisition of image feature amount of image and associatedimage processing are performed by the computer 21 with an imageprocessing program corresponding to the flowchart shown in FIG. 42. Theflowchart shown in the figure corresponds to the preceding stage in theimage processing program, and processing is performed to sum image dataaccording to plural evaluation criteria.

[0309] According to this embodiment, the case will be described wheretwo evaluation criteria are used. Common points are that in both cases,not all picture elements are considered, picture elements are sampledaccording to predetermined criteria, and luminance is summed for thesampled picture elements. What is different is that in one method,picture elements are sampled uniformly, whereas in the other method,edge picture elements are selected for sampling. Luminance summationresults are described hereafter, but here it should be noted that pluralfeature amounts are obtained according to different evaluation criteriaby changing the sampling method.

[0310] Uniform sampling means that luminance is summed for all pictureelements in the image, and the luminance distribution is determined forthe image as a whole. A feature amount is thereby obtained which isuseful as a reference when a scenic photograph is dark overall orcontrast is narrow. In the other method, as edge picture elements are asharp part of the image, the luminance is summed for picture elementsrelated to the object in the image. Therefore provided that the objectis sufficiently light even if the background is dark, a feature amountis obtained which is useful as a reference when the image issufficiently light. According to this embodiment, these feature amountsare selected automatically according to the image processing method.

[0311] Referring to the flowchart of FIG. 42, in this summationprocessing, object picture elements are main scanned in the horizontaldirection and auxiliary scanned in the vertical direction for image datacomprising dot matrix picture elements, and displaced, as shown in FIG.24, and the summation is performed by determining whether or not eachpicture element scanned is to be included in the sampling.

[0312] First, in a step SC110, the edginess of each picture element isdetermined. Specifically, the determination of edginess may be the sameas that of the aforesaid step SA110.

[0313] In the step SC120, the edginess is compared with the samethreshold value, and it is determined whether or not the variation islarge. If as a result of comparison it is determined that edginess islarge, it is determined that this picture element is an edge pictureelement, and the image data for the picture element is sampled in a stepSC130 and stored in the work area. The work area may be a RAM in thecomputer 21, or it may be a hard disk 22.

[0314] In this embodiment, the object is sampled based on edginess, butof course the method of sampling the object is not limited to this. Forexample, the chromaticity of each picture element may be found, andpicture elements for which the chromaticity is within a predeterminedrange can be sampled as the object.

[0315] Specifically, the processing of the step SC110 and step SC120 issubstituted by the processing of a step SC115 and a step SC125respectively, the chromaticity for each picture element is calculated inthe step SC115, and in the step SC125, it is determined whether or notthe x-y chromaticity that was converted based on RGB gradation data foreach picture element is within the range of skin color. If it is skincolor, the image data of the picture element is sampled in the stepSC130 and also stored similarly to work area.

[0316] On the other hand, in parallel with the aforesaid determinationof edginess, it is determined in a step SC140 whether or not thispicture element is to be sampled by uniform sampling. Uniform samplingis identical to the above. In the step SC140 it is determined whether ornot the picture element is to be sampled, and if so, the picture elementis sampled in a step SC150. It will be understood that the sampling ofimage data in the steps SC130, SC150 means the summation of luminancebased on this image data.

[0317] To perform this processing for all picture elements of the imagedata, a picture element to be processed is displaced in a step SC160,and the processing is repeated until it is determined in a step SC170that processing of all picture elements has finished.

[0318] Thereafter, a feature amount is obtained by using the summationresult Dist_ave obtained by uniform sampling and the summation resultDist_edg obtained by edge picture element sampling according to theintended image processing method, and so optimum image processing basedon this feature amount may be performed. As an example, FIG. 43 shows aflowchart for performing expansion of contrast and lightnesscompensation. The basic method of increasing contrast according to thisembodiment is as described hereabove. The correction parameters aregenerated in steps SC310-SC330. Then, it is determined in a step SC340whether contrast correction and lightness compensation are necessary,and if it is determined that they are, conversion of image data isperformed in a step SC350. In this step SC350, reference is made to aconversion table corresponding to equations (14)-(16) for image data(R0, G0, B0) for all picture elements, and the process of obtainingimage data (R, G, B) after conversion is repeated.

[0319] However in this case the luminance summation results are used asappropriate, and a feature amount is obtained comprising both ends andthe median of the luminance distribution. Contrast correction andlightness compensation are applied using this feature amount, but thisis not the only specific example of image processing, and there are alsovarious feature amounts which may be used. For example, saturationemphasis image processing may be performed as shown in FIG. 20.

[0320] In this case, the evaluation unit comprises a first stage of aprogram for summing image data by the uniform sampling method (stepSA410) up to the acquisition of a feature amount which is a saturationemphasis indicator S (step SA420), and hardware for implementing thisfirst stage of the program. The image processing unit comprises thelatter stage of the program for performing conversion of image data(step SA440), and hardware for implementing this latter stage of theprogram.

[0321] Regarding contrast correction, lightness compensation andsaturation emphasis, it is determined whether to perform imageprocessing in each case. However, it is not absolutely necessary to makea choice as to whether or not to perform image processing. Specifically,a degree of emphasis degree is set for each, and image processing may beperformed with the set degree of emphasis.

[0322] Next, the operation of this embodiment having the aforesaidconstruction will be described.

[0323] A photographic image is read by the scanner 11, and printed bythe printer 31. Specifically, when the operating system 21 a is runningon the computer 21, the image processing application 21 d is started,and reading of the photograph is started by the scanner 11. After theread image data has been assimilated by the image processing application21 d via the operating system 21 a, the picture elements to be processedare set in initial positions. Next, the edginess is determined based onequations (1)-(3) in the step SC110, and the edginess is compared with athreshold value in the step SC120. When the edginess is large, it isdetermined that the picture element to be processed is an edge pictureelement, and image data for the corresponding picture element is storedin the work area in the step SC130. In the step SC140, it is determinedwhether or not picture elements to be processed are to sampleduniformly, and if so, in the step SC150, image data for these pictureelements is stored in the work area. The above processing is repeateduntil it is determined to have been performed for all picture elementsin the step SC170 while picture elements to be processed are displacedin the step SC160.

[0324] According to this embodiment, image data was stored in the workarea, but from the viewpoints of memory capacity and processing time, itis not absolutely necessary to store image data itself in the work area.Specifically, histograms of luminance distribution or saturationsubstitute value distribution for picture elements determined to bethose of the object are generated, and histogram information maytherefore be pre-stored in the steps SC120, SC150.

[0325] When summation has been performed on all picture elements,luminance distribution histograms are found for the summation resultDist_ave obtained by uniform sampling and the summation result Dist_edgobtained by edge picture element sampling in the step SC310. A parameterfor expansion processing is determined based on equations (8), (9) inthe step SC320, and a parameter for luminance compensation is determinedbased on equations (10)-(13) in the step SC330. These parameters arecompared with a predetermined threshold value in the step SC340, and ifit is determined that image processing should be performed, theluminance is converted based on the aforesaid parameter in the stepSC350. In this case, the luminance conversion table shown in FIG. 16 maybe generated to reduce the computing amount, and image data may beconverted based on equations (14)-(16).

[0326] Subsequently, the processed image data may be displayed on thedisplay 32 via the display driver 21 c, and if it is satisfactory, itmay be printed by the printer 31 via the printer driver 21 b.Specifically, the printer driver 21 b inputs RGB gradation data withemphasized edges, performs rasterization corresponding to the print headarea of the printer 31 after a predetermined mapping, color convertsrasterized data from RGB to CMYK, converts CMYK gradation data to binarydata, and outputs it to the printer 31.

[0327] Due to the above processing, photographic image data read via thescanner 11 is automatically subjected to the optimum contrast correctionand lightness compensation, displayed by the display 32, and printed bythe printer 31. Specifically, plural feature amounts are obtained usingplural evaluation criteria, and optimum image processing is performedwith different feature amounts according to the image processing methodsof contrast correction or lightness compensation.

[0328] However, the invention is not limited to this contrast correctionand luminance compensation. In the case also of saturation emphasis, afeature amount is acquired by sampling saturation according to asuitable criterion depending on the saturation emphasis processing, andimage processing is performed based on this feature amount. In this way,optimum image processing is performed.

[0329] Hence, the computer 21 which is the core of image processingcalculates the luminance distribution based on the summation resultssampled according to different evaluation criteria in the step SC310,obtains different feature amounts from different luminance distributionhistograms in the steps SC320 and SC330, and converts image data basedon these feature amounts in the step SC350. This permits optimum imageprocessing.

Embodiment 4

[0330] Next, an embodiment will be described concerned mainly with imageevaluation which is an indicator for image processing.

[0331]FIG. 44 is a block diagram of an image processing system whichperforms image processing by implementing an image evaluation methodaccording to one embodiment of this invention. A typical hardwareconstruction is similar to the system shown in FIG. 2.

[0332] In FIG. 44, the image reader 10 outputs photographic image dataas dot matrix picture elements to an image processing apparatus 20D. Theimage processing apparatus 20D calculates an evaluation result bysumming the image data after predetermined processing, determines thecontent and extent of image processing based on the evaluation result,and then performs image processing. The image processing apparatus 20Doutputs the processed image data to an image output device 30, and theimage output device outputs the processed image as dot matrix pictureelements.

[0333] The image processing apparatus 20D sums the image databeforehand, and calculates an evaluation result for the correspondingimage. The image data are summed individually using plural evaluationcriteria, and are combined by varying the weighting according topredetermined conditions. Therefore, the image processing apparatus 20Dis an image data evaluation unit.

[0334] Image evaluation and associated image processing are performed inthe computer 21 by an image processing program corresponding to aflowchart such as is shown in FIG. 45. The flowchart shown in the figurecorresponds to the first stage of image evaluation in the imageprocessing program, and processing is performed to obtain apredetermined evaluation result by summing the image data according toplural evaluation criteria. The basic method of image evaluation issubstantially identical to that described above, however in thefollowing description, a simple evaluation result is obtained by usingimage evaluation options.

[0335] According to this embodiment, two of the evaluation criteria usedwill be described. Common points are that in both cases, pictureelements are thinned according to a predetermined criterion instead ofconsidering the whole image, and the luminance of the sampled pictureelements is summed.

[0336] A difference is that whereas in one case, picture elements aresampled uniformly, in the other case, edge picture elements areselected. The luminance summation results are described hereafter, butthe image evaluation can be changed by changing the sampling method inthis way. Uniform sampling of picture elements means that the luminanceis summed for all picture elements in the image, and the luminancedistribution is determined for the image as a whole. The evaluation istherefore useful as a reference when a scenic photograph is dark overallor contrast is narrow. In the other method, as edge picture elements area sharp part of the image, the luminance is summed for picture elementsrelated to the object in the image. For example, provided that theobject is sufficiently light even if the background is dark, anevaluation result is obtained where the image is sufficiently light.According to this embodiment, the image is determined by suitablycombining two evaluation criteria, i.e. a criterion selected by theoperator or automatic processing.

[0337] Referring now to the flowchart of FIG. 45, in steps SD110-SD170,image data comprising dot matrix picture elements is main scanned in thehorizontal direction and auxiliary scanned in the vertical direction asshown in FIG. 24 as described in the aforesaid embodiment, and imagedata for edge picture elements and uniformly sampled picture elements isstored in the work area.

[0338] After luminance is summed for all picture elements concerned bythese various sampling methods, image evaluation options are input in astep SD180. FIG. 46 shows an image evaluation option input screendisplayed on the display 32. Three choices are available: portrait,scenic photograph and automatic setting.

[0339] As shown in FIG. 47, weightings must be adjusted in order togenerate a histogram to evaluate the image which is obtained bycombining a luminance histogram obtained by uniform sampling and aluminance histogram obtained by edge sampling.

[0340] When the weighting coefficient k is adopted, a summation resultDist_Sum for evaluation is obtained from the uniform sampling summationresult Dist_ave and the edge sampling summation result Dist_edg by therelation:

Dist_Sum=k×Dist_edg+(1−k)×Dist_ave   (68).

[0341] The closer the weighting coefficient k approaches “0” the morethe whole image is emphasized, and the closer it approaches “1”, themore the object in the photograph is emphasized.

[0342] As a result, after setting the options on the image evaluationoption input screen shown in FIG. 46, the routine branches depending onthe option in a step SD190. When a portrait is selected, k is set to 0.8in a step SD192, and when a scenic photograph is selected, k is set to0.2 in a step SD194.

[0343] The remaining option is the “AUTO” setting. In this auto setting,based on the edge picture elements sampled as described above, the imagemay be considered as a scenic photograph and the weighting coefficientapproaches “0” when there are few edge picture elements, while the imagemay be considered as a portrait and the weighting coefficient approaches“1” when there are many edge picture elements By using the samplingnumber x_edg of edge picture elements and the uniform sampling numberx_ave, the weighting coefficient may be computed from:

k−x_edg/(x_edg+x_ave)   (69)

[0344] in a step SD196, and the summation result Dist_Sum used forevaluation can be obtained.

[0345] After the summation result Dist_Sum used for evaluation isobtained, image evaluation can, be performed. Of course, a furtherdetermination may be made using this summation result, and it maybasically be varied as necessary depending on the image processing whichuses the summation result.

[0346] Subsequently, the optimum image processing is determined based onthe summation results and performed. As an example, FIG. 48 shows aflowchart to perform expansion of contrast and luminance compensation.In a basic technique to increase contrast according to this embodiment,the luminance distribution is found based on image data as in theaforesaid embodiment, and if this luminance distribution uses only partof the actual gradation width (255 gradations), it is expanded.

[0347] Therefore, in a step SD310, a histogram of luminance distributionis generated as the summation result Dist_Sum from the weightingcoefficient k, and the expansion width is determined in a step SD320.The expansion width determining method is as described above. Thegeneration of a predetermined conversion chart corresponds to theexpansion width determination processing of the step S320, so image datacan be modified by referring to this chart. By expanding the luminancerange, not only is the contrast emphasized, but it is also very usefulto adjust the luminance range at the same time. Therefore the luminanceof the image is determined in a step SD330, and a parameter is generatedfor the correction.

[0348] In this case, the summation results used for determining theimage are used as evaluation criteria. Contrast correction and luminancecompensation are performed, but specific examples of image processingare not limited to this, and therefore various summation results may beused as evaluation criteria.

[0349] Examples of image processing are saturation emphasis or edgeemphasis. In these cases, image vividness and image sharpness need to beevaluated. The summation may be performed by the above methods.

[0350] It may be determined whether image processing is performedregarding contrast correction, luminance compensation, saturationemphasis and edge emphasis, but it is not absolutely necessary to make achoice as to whether or not to perform image processing. Specifically, adegree of emphasis is set for each, and image processing is performedusing this set degree of emphasis.

[0351] Next, the action of this embodiment having the aforesaidconstruction will be described.

[0352] Assume that a photographic image is read with the scanner 11, andprinted by the printer 31. When the operating system 21 a is running,the image processing application 21 d is started by the computer 21, andreading of a photograph is started relative to scanner 11. After theread image data has been assimilated via the operating system 21 a by animage processing application 21 d, the picture elements to be processedare set in initial positions. Next, the edginess is determined based onequations (1)-(3) in the step SD110, and the edginess is compared with athreshold value in the step SD120. When the edginess is large, it isdetermined that the picture element to be processed is an edge pictureelement, and image data for the corresponding picture element is storedin the work area in the step SD130. In the step SD140, it is determinedwhether or not picture elements to be processed are to sampleduniformly, and if so, in the step SD150, image data for these pictureelements is stored in the work area. The above processing is repeateduntil it is determined to have been performed for all picture elementsin the step SD170 while picture elements to be processed are displacedin the step SD160.

[0353] When this has been performed for all picture elements, image datasampled according to different evaluation criteria is stored indifferent work areas, and in a step SD180, image evaluation options areinput. The operator, looking at the image, may select either a portraitor scenic photograph is that can be determined. If it cannot bedetermined or when it is desired to fully automate operations, theoperator selects the auto setting. When a portrait is selected, theweighting coefficient k becomes “0.8”, which puts more weight on thesummation results for the edge picture elements. When a scenicphotograph is selected, the weighting coefficient k becomes “0.2”, whichputs more weight on uniformly sampled summation results. When the autosetting is selected, the weighting coefficient k is set according to theproportion of edge picture elements. However, in any of these casesplural evaluation criteria are adopted using the weighting coefficientk. This permits a flexible evaluation not limited to just one criterion.

[0354] According to this embodiment, image data was stored in the workarea, but from the viewpoints of memory capacity and processing time, itis not absolutely necessary to store image data itself in the work area.Specifically, histograms of luminance distribution or saturationsubstitute value distribution for sampled picture elements aregenerated, and histogram information may therefore be prestored in thesteps SD120, SD150.

[0355] When contrast correction and luminance compensation are performedautomatically, a weighting coefficient is used, a histogram of luminancedistribution is found in the steps SD120, SD150, SD310, a parameter forexpansion processing is determined based on equations (8), (9) in thestep SD320, and a parameter for luminance compensation is determinedbased on equations (10)-(13) in the step SD330. These parameters arecompared with a predetermined threshold value in a step SD340, and if itis determined that image processing should be performed, the luminanceis converted based on the aforesaid parameter in a step SD350. In thiscase, a luminance conversion table shown in FIG. 16 may be generated toreduce the computing amount, and image data may be converted based onequations (14)-(16).

[0356] Subsequently, the processed image data may be displayed on thedisplay 32 via a display driver 21 c, and if it is satisfactory, it isprinted by the printer 31 via a printer driver 21 b. In other words theprinter driver 21 b inputs RGB gradation data with emphasized edges,performs rasterization corresponding to the print head area of theprinter 31 after a predetermined mapping, saturation converts rasterizeddata from RGB to CMYK, converts CMYK gradation data to binary data, andoutputs it to the printer 31.

[0357] The photographic image data read via the scanner 11 and printedby the printer 31 or displayed by the display 32, is thereforeautomatically subjected to optimum contrast correction and luminancecompensation. More specifically, the image can be determined moreflexibly by adopting plural evaluation criteria, and optimum imageprocessing realized by contrast correction and luminance compensation.

[0358] However, the invention is not limited to this contrast correctionand luminance compensation. Saturation and edginess are also sampled byplural evaluation criteria and summed, and weighting coefficients areadjusted and added. Therefore, image processing is performed after aflexible determination which is not limited to a single evaluationcriterion.

[0359] In this way, the computer 21 which is the core of imageprocessing first samples image data for picture elements according todifferent evaluation criteria in the steps SD120, SD140, and determinesthe weighting coefficient k in the steps SD192-SD196 based on imageevaluation options input in the step SD180. It then generates aluminance distribution histogram by adding summation results in the stepSD310 using these weighting coefficients, and performs optimum imageprocessing in the steps SD310-SD350.

We claim:
 1. An image processing apparatus for inputting photographicimage data comprising dot matrix picture elements and performingpredetermined image processing thereon, said apparatus comprising: animage data acquiring unit which acquires said photographic data, animage processing indicator specifying unit which performs predeterminedsummation processing on all picture elements based on said acquiredimage data and specifying an image processing indicator, and aprocessing unit which determines image processing contents based on saidspecified indicator and performs image processing, wherein said imageprocessing indicator specifying unit comprises an object determiningunit which determines picture elements having a large image variationamount to be those of the object, and said processing unit determinesimage processing contents based on image data for picture elementsdetermined to be those of the object and performs image processing basedon said determined contents.
 2. An image processing apparatus as definedin claim 1, wherein in said object determining unit, said imagevariation amount is determined based on a difference of image databetween adjacent picture elements.
 3. An image processing apparatus asdefined in claim 1, wherein in said object determining unit, a criterionfor determining whether or not said image variation amount is large ismade to vary according to a position of said image.
 4. An imageprocessing apparatus as defined in claim 3, wherein in said objectdetermining unit, said criterion is set lower for the center part thanfor the periphery of said image.
 5. An image processing apparatus asdefined in claim 3, wherein in said object determining unit, saidcriterion is determined based on a distribution of said image variationin each part of the image.
 6. An image processing method for inputtingphotographic image data comprising dot matrix picture elements andperforming predetermined image processing thereon, said methodcomprising: an image data acquiring step for acquiring said photographicdata, an image processing indicator specifying step for performingpredetermined summation processing on all picture elements based on saidacquired image data and specifying an image processing indicator, and aprocessing step for determining image processing contents based on saidspecified indicator and performing image processing, wherein said imageprocessing indicator specifying step comprises an object determiningstep in which picture elements having a large image variation amount aredetermined to be those of the object, and in the aforesaid processingstep, image processing contents are determined based on image data forpicture elements determined to be those of an object and imageprocessing is performed based on the determined contents.
 7. A recordingmedium for recording an image processing control program which inputsphotographic image data comprising dot matrix picture elements andperforms predetermined image processing thereon, said programcomprising: an image data acquiring step for acquiring said photographicdata, an image processing indicator specifying step for performingpredetermined summation processing on all picture elements based on saidacquired image data and specifying an image processing indicator, and aprocessing step for determining image processing contents based on saidspecified indicator and performing image processing, wherein said imageprocessing indicator specifying step comprises an object determiningstep in which picture elements having a large image variation amount aredetermined to be those of an object, and in said processing step, imageprocessing contents are determined based on image data for pictureelements determined to be those of the object and image processing isperformed based on said determined contents.
 8. An image processingapparatus for inputting photographic image data comprising dot matrixpicture elements and performing predetermined image processing thereon,said method comprising: an image data acquiring unit which acquires saidphotographic data, an image processing indicator specifying unit whichperforms predetermined summation processing on all picture elementsbased on said acquired image data and specifies an image processingindicator, and a processing unit which determines image processingcontents based on said specified indicator and performs imageprocessing, wherein said image processing indicator specifying unitcomprises a feature amount uniform sampling unit which determines animage processing intensity by uniformly sampling a feature amount over awhole screen, and a feature amount weighting reevaluation unit whichreevaluates said feature amount sampled in said feature amount samplingunit with a predetermined weighting, and in said processing unit, saidimage processing intensity is determined based on said reevaluatedfeature amount, and image processing is performed with said determinedintensity.
 9. An image processing apparatus as defined in claim 8,wherein in said feature amount uniform sampling unit, said featureamount is sampled for picture elements selected by uniformly thinningall picture elements according to a predetermined criterion.
 10. Animage processing apparatus as defined in claim 8, wherein in saidfeature amount uniform sampling unit, a feature amount is sampled inarea units obtained by dividing said image according to a predeterminedcriterion, and in said feature amount weighting reevaluation unit, aweighting is set for each of said areas so as to reevaluate said featureamount.
 11. An image processing apparatus as defined in claim 8, whereinin said feature amount weighting reevaluation unit, said weighting isvaried according to a positional relationship determined by the positionof each picture element in the image.
 12. An image processing apparatusas defined in claim 8, wherein in said feature amount weightingreevaluation unit, an image variation amount is calculated, and saidweighting is increased in a part where said image variation amount islarge.
 13. An image processing apparatus as defined in claim 8, whereinin said feature amount weighting reevaluation unit, a chromaticity ofeach picture element is calculated, a number of picture elements havinga chromaticity within a chromaticity range of a target for which it isdesired to sample a feature amount is calculated, and said weighting isincreased in a part where said number of picture elements is large. 14.An image processing apparatus as defined in claim 8, wherein in saidfeature amount weighting reevaluation unit, temporary weightingcoefficients are calculated separately based on plural factors, and saidweighting coefficients are summed according to their degree ofimportance so as to give a final weighting coefficient which is thenapplied.
 15. An image processing apparatus for inputting photographicimage data comprising dot matrix picture elements and performingpredetermined image processing thereon, said apparatus beingcharacterized in comprising: an image data acquiring unit which acquiressaid photographic data, an image processing indicator specifying unitwhich performs predetermined summation processing on all pictureelements based on said acquired image data and specifying an imageprocessing indicator, and a processing unit which determines imageprocessing contents based on said specified indicator and performs imageprocessing, wherein said image processing indicator specifying unitcomprises an evaluation unit which obtains a feature amount by inputtingsaid photographic data, summs the image data for all picture elementsand obtains feature amounts according to plural predetermined evaluationcriteria, and in said processing unit, the image data can be convertedby plural techniques and the feature amounts obtained by said evaluationunit are used according to each of said techniques.
 16. An imageprocessing apparatus as defined in claim 15, wherein said evaluationunit comprises a standard for sampling an object in a photographic imageand summing image data for object picture elements so as to obtain afeature amount, and in said processing unit, a feature amount obtainedfrom object picture elements is used when a feature amount for a centerpart of said image data is used in one image processing technique. 17.An image processing method for inputting photographic image datacomprising dot matrix picture elements and performing predeterminedimage processing thereon, said method comprising: an image dataacquiring step for acquiring said photographic data, an image processingindicator specifying step for performing predetermined summationprocessing on all picture elements based on said acquired image data andspecifying an image processing indicator, and a processing step fordetermining image processing contents based on said specified indicatorand performing image processing, wherein said image processing indicatorspecifying step comprises an evaluation step for inputting saidphotographic data, summing the image data for all picture elements, andobtaining feature amounts according to plural predetermined evaluationcriteria, and in said processing step, the image data can be convertedby plural techniques and the feature amounts obtained in said evaluationstep are used according to each of said techniques.
 18. A medium forrecording an image processing control program for inputting image datacomprising dot matrix picture elements and performing predeterminedimage processing thereon, said program comprising: an image dataacquiring step for acquiring said photographic data, an image processingindicator specifying step for performing predetermined summationprocessing on all picture elements based on said acquired image data andspecifying an image processing indicator, and an image processing stepfor determining image processing contents based on said specifiedindicator and performing image processing, wherein said image processingindicator specifying step comprises a step for inputting image data forsaid photographic data, summing the image data for all picture elements,and obtaining a feature amount according to plural predeterminedevaluation criteria, and in said image processing step, the image datacan be converted by plural techniques and the feature amount obtainedare used according to each of said techniques.
 19. An image evaluationdevice, comprising: an image data input unit which inputs photographicimage data comprising dot matrix picture elements, and an image dataevaluation unit which sums image data for all picture elements accordingto a predetermined criterion, comprises plural evaluation criteria whichare applied to said summation results to evaluate the image based onsaid results, and combines said evaluation results according to apredetermined weighting based on each of said evaluation criteria.
 20. Arecording medium on which an image evaluation program is recorded, saidprogram comprising the steps of: photographic image data comprising dotmatrix picture elements being input by a computer, image data for allpicture elements being summed according to a predetermined criterion,and the image being evaluated based on the summation results, whereinsaid program comprising plural criteria for evaluating said summationresults, and combining said evaluation results according to apredetermined weighting based on each of said evaluation criteria. 21.An image evaluation method comprising the steps of: photographic imagedata comprising dot matrix picture elements being input, image data forall picture elements being summed according to a predeterminedcriterion, and the image being evaluated based on the summation results,wherein said method comprising plural criteria for evaluating saidsummation results, and said results are combined according to apredetermined weighting based on each of said evaluation criteria.